new /weights endpoint

[andrewm4894]

Following some internal discussion we have decided to introduce a new api/v1/weights endpoint to the agent that will be a more general endpoint to take in some input params and return, in general, a score or "weight" for each dimension. Generally this scrore or weight would be some sort of scoring for ranking, filtering or ordering dimensions, charts and contexts.

This will be able to serve existing metric correlation features and anomaly advisor features. Once this is done we should be able to retire the special anomaly_detection.anomaly_rates chart. api/v1/metric_correlations can also be replaced with api/v1/weights.

@vkalintiris @MrZammler @hugovalente-pm @ktsaou @amalkov fyi

@amalkov or @ktsaou i cant quite recall exactly what the proposed schema was for the new api/v1/weights endpoint, when possible can you stick in what we briefly landed at.

I just wanteed to make sure we have a wider discussion and everyone aware as this will affect both AA, single node MC, and be a blocker before we can tackle cross node MC.

[hugovalente-pm]

I'm not sure we closed the entire schema, I know we delved on the common part of it that would be something like:

{
        ...
	"weighted_contexts": {
		"context-name-1": {
			"chart-name-1": {
				"dimensions": {
					"dim1.1": 0.9872,
					"dim2.1": 0.7842
				}
			},
			"chart-name-2": {
				"dimensions": {
					"dim2.1": 0.8789,
					"dim2.2": 0.5421,
					"dim2.3": 0.4012
				}
			}
		},
		"context-name-2": {
			"chart-name-4": {
				"dimensions": {
					"dim4.1": 0.9872
				}
			}
		}
	}
}

I guess there will still be specific sections in the response that relate to the methods for weighting that was used, e.g. on metric correlations response we have

{
	"after": 1659015972,
	"before": 1659016571,
	"duration": 599,
	"points": 500,
	"baseline_after": 1659015373,
	"baseline_before": 1659015972,
	"baseline_duration": 599,
	"baseline_points": 500,
	"statistics": {
		"query_time_ms": 506.050000,
		"db_queries": 1540,
		"db_points_read": 4314882,
		"query_result_points": 4357920,
		"binary_searches": 6163728
	},
	"group": "average",
	"method": "ks2",
	"options": "nonzero,null2zero,unaligned",
	"correlated_charts": {
		... 
	},
	"correlated_dimensions": 776,
	"total_dimensions_count": 4208
}

[amalkov]

@andrewm4894 @hugovalente-pm what are we going to do with this ticket?

[andrewm4894]

this discussion can be closed as implemented - ill cross link any public issues related to this in here.

will also add public api docs once we have them later down the road.

[andrewm4894]

actually - i think we just do nothing with this discussion and leave it as is as i dont think you close discussions in GH discussions :)

[hugovalente-pm]

if it was a question you could mark it as answered, but since it is a discussion I think we just leave it