Add pre-release checks based on netdata

[vkalintiris]

Main idea:

1. Run netdata in a container,
2. Observe CPU, RAM, disk, network (?) consumption of the agent|system after X minutes,
3. Setup a pre-release nightly CI job that runs steps 1 & 2.
4. Fail the job if any of the observed values is Y times above the ones in the original run.

I think that with this process, the CI check would ensure the agent is not broken in a fundamental way for nightly releases.

The reason for opening this discussion is that a while ago we merged 3 PRs that:

• Consumed a lot of disk space in a short amount of time,
• Used a single-core 100% of the time,
• Crashed the agent on startup.

@Ferroin WDYT?

[Ferroin]

The viability of this kind of depends on how big of a sampling window we want to use. It’s probably reasonably doable with, say, a five minute sampling window, but much more than that is probably not realistic.

There are two other limitations I see to such checks:

• We would want to run this check on PRs as well, not just as part of the release prep.
• This can only realistically check a default configuration in a relatively minimalistic system.

I don’t really consider any of those blockers though, and can probably put something like this together relatively quickly.

[vkalintiris]

We would want to run this check on PRs as well, not just as part of the release prep

Ideally, yes. However, we need to let this run for a long enough time to be able to catch possible problems. Would it be okay to start with something like 20 mins and run it only before creating a new nighlty?

[Ferroin]

Because of how the agent works, we can’t (reliably) get a solid indication of a memory leak unless we run long enough to completely fill the dbengine page cache, which will need to be a very long run given the relatively small number of metrics we would be looking at in a GHA runner (or a Docker container that is properly isolated from the host). If we want some assurance there, we would need to be looking at a run of an hour or more at minimum, possibly even far longer, and even for nightlies that’s currently impractical (as it is, publishing nightlies takes at least 2.5 hours, which is far too long).

Given this, I see no benefit to running for 20 minutes for this check. I think we’re going to be better off running for something like 5 minutes and then comparing the difference in memory usage since the start instead of the absolute memory usage and fail if that is too large a number.

For the CPU check, we shouldn’t need much longer than that either, and can actually scrape Netdata itself to check this if we do things correctly.

For the disk check what matters is, like with RAM, the rate of change and not the absolute state, so we again don’t need a super-long sampling window.

[vkalintiris]

Thanks @Ferroin!

I don't worry so much about the actual tests/checks that we can run at this point. The intention is to add them (or modify the agent) incrementally over time.

If we want some assurance there, we would need to be looking at a run of an hour or more at minimum, possibly even far longer, and even for nightlies that’s currently impractical (as it is, publishing nightlies takes at least 2.5 hours, which is far too long)

So what you're saying is that the CI job can run up to 2.5 hours :D

Joking aside, I'm curious, could we run this job in parallel with the nightly job(s)/task(s) and have it block the release on failure?

Given this, I see no benefit to running for 20 minutes for this check

To be honest, I believe we could get away with less than 20 minutes. The reason I mentioned that duration is because it takes at least 1024 ~= 17 mins for DBEngine to flush the 1st set of pages to disk.

[Ferroin]

Joking aside, I'm curious, could we run this job in parallel with the nightly job(s)/task(s) and have it block the release on failure?

Not easily unfortunately. It’s remarkably complicated to reliably block on completion of a GHA workflow without handling it as a normal dependency. The most realistic way to handle this would be to give it a workflow_call trigger and have the release workflow call it as part of the release preparation process (and also have push and pull_request triggers so it runs on PRs and merges).

To be honest, I believe we could get away with less than 20 minutes. The reason I mentioned that duration is because it takes at least 1024 ~= 17 mins for DBEngine to flush the 1st set of pages to disk.

Makes sense. I would think we can probably avoid the issues of needing to wait on the dbengine quite as long if we intentionally set the thresholds low for it, but for the moment I think we’re better off just starting with a shorter run and looking at the average rate of change of RAM over the course of the run instead of the absolute amount of RAM (at least, for checking for excessive memory usage we probably are).