Enabling ML by default

[shyamvalsan]

Hey @andrewm4894 @vkalintiris @ktsaou @cpipilas

Creating this discussion for us to get to the immediate actions required to enable ML by default. Please share your feedback (and add others you think should be on this discussion thread).

Let's try to arrive at the list of actions by 16-Feb-2022

Objective

• Enable ML by default. This implies enabling anomaly detection by default for all possible metrics, metric correlation continues to be on-click-ML triggered by user.
• Ensure that resource and performance utilization remain within acceptable limits when ML is enabled on the edge.
• Choose the right default configuration parameters for ML to safely allow it to run everywhere all the time.

[ktsaou]

• Enable ML by default. This implies enabling anomaly detection by default for all possible metrics, metric correlation continues to be on-click-ML triggered by user

Metric correlation is not related at all to ML. Currently it is also centralized, so as far as the agents are concerned, the current metric correlation is concerned, it is just /api/v1/data queries. Right?

• Ensure that resource and performance utilization remain within acceptable limits when ML is enabled on the edge

We need to test this. Any ideas what tests we should run?

• Choose the right default configuration parameters for ML to safely allow it to run everywhere all the time

What are the current settings?

[shyamvalsan]

@vkalintiris I think we need to do a POC with an online approach to answer that for sure, maybe using something like a Mini Batch K means - another option would be to measure approximate entropy for each metric and have a threshold within which we think Anomaly Detection is valuable.

[andrewm4894]

Is there anything else that influences cpu or memory consumption?

charts to skip from training is the main one - this is why the defaults here is like an allow list of typical default charts. Users would then need to add to it if wanted to expand to more charts.

[ktsaou]

charts to skip from training is the main one - this is why the defaults here is like an allow list of typical default charts. Users would then need to add to it if wanted to expand to more charts.

hm... ideally it should be automatic.
This is why I suggested to eliminate all the ones that are not useful for ml.

[vkalintiris]

How many metrics can we exclude with this? There will be many metrics that are just constant... Can we find out how many?

I built an agent and let it run overnight on an idle machine. I used the /data endpoint on every chart to figure out the number of constant dimensions. They are in the range of 40%-50%.

If a metric is totally constant (i.e. min = max for the whole duration), why to train a model for it?

1. This optimization does not apply to the 1st training iteration. We'd have to build a model for the constant dimensions, because subsequent values might differ. As long as the values remain constant, we can skip retraining.
2. Somewhat obvious but for completeness sake - the peak CPU usage is not reduced. Only the total CPU consumption over the training window.

My intention is to:

1. Confirm the results we got from the /data endpoint, by checking directly the internal storage numbers of each dimension.
2. Implement the optimization, compare it against the current implementation, and test it on staging.

First, I'll prioritize addressing review comments in #12083, in order to unblock Andrew and the frontend team.

I'll share any updates here.

[shyamvalsan]

Thanks @vkalintiris this sounds like a good plan of action.

And about this

This optimization does not apply to the 1st training iteration. We'd have to build a model for the constant dimensions, because subsequent values might differ. As long as the values remain constant, we can skip retraining.

just thinking out aloud, can we not have a 0th iteration where we do not train the model but just observe and exclude constant dimensions? And then training starts in the next iteration?

[andrewm4894]

For me the following steps would need to happen before we consider enabling by default.

1. anomaly rate on any chart implemented on frontend.
2. one month of full active, internal, company wide, dogfooding.
3. after internal dogfooding, one month of usage and feedback by early adopters in the netdata community who have enabled the feature.
4. some rules around conditions that have to be met by the node for this feature to be activated, otherwise it should stay default off.

[ktsaou]

@andrewm4894 what do you expect to get out of these? What are you concerned about and how these steps will help?

I am interested about 2 and 3.

I think 1 will be implemented in the mean-time.
And 4 is accurate. We have to figure out these conditions.

But as I understand it, the first and key step to proceed further, should be understanding the CPU impact this has. Without it, we are just theorizing...

[andrewm4894]

1,2,3 are all just about getting as much exposure as possible to get as much qualitative feedback on the actual ml in a wide range of nodes and settings. From that we build our knowledge of where it works well, and where its maybe weaker and we need to improve things under the hood a little more. Then we would have a more informed viewpoint to decide if its "ready" or "good enough" for enabling by default as the first generation of the models.

[ktsaou]

I think that if the cpu impact is not significant, we should release it enabled by default as soon as possible. It is better to interact with real users instead of keeping it for us for so long. The number of use cases that will be revealed from users is enormous compared to the ones we will find.

So, the key question is what is the impact. We should do the most to minimize the impact. This is why intelligently selecting the metrics to be trained is very important.

[shyamvalsan]

To sum up the discussion.

This feature is planned for early Q2 along with "anomaly rate on every chart" this gives us a month or more of internal dogfooding and hopefully a few weeks of early adopter testing as well.

In the mean time we can answer the question about the impact and how to minimize it.

[ktsaou]

@shyamvalsan @cpipilas please open a ticket about measuring the impact. We need hard facts. So, someone needs to work on this asap. Please discuss with @stelfrag to figure out who can help.

@shyamvalsan there is another interesting finding in this discussion. The exclusion list for metrics is wrong. It excludes all the applications by default and includes many metrics that are just constant over time. We need to find a solution for this. I would suggest to open another ticket to discuss it and come with reasonable solutions.

[shyamvalsan]

Opened a new ticket #12173 specifically to measure the impact.

[shyamvalsan]

Also created a couple of tickets to cover the automating selection of metrics to train anomaly detection for.

#12177 Ensure that "charts to skip from training" excludes only charts that anomaly detection will not add value
#12178 [POC] Identify an optimal method to auto select what dimensions to train anomaly detection on

And once the POC results are available we can create a separate ticket to actually make the change on the agent side.

cc: @ktsaou @cpipilas