Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v11.0.x] Alerting: Fix scheduler to sort rules before evaluation #88021

Merged
merged 2 commits into from
May 20, 2024

Conversation

grafana-delivery-bot[bot]
Copy link
Contributor

Backport 05d6813 from #88006


What is this feature?
This PR fixes Grafnaa Alerting scheduler to sort the rules that are scheduled for the current tick before spreading them out.

Why do we need this feature?
This is needed to guarantee that rule is consistently evaluated every X interval. The order of items scheduled for evaluation is important because the index of element is used to calculate a delay. That delay can be up to 10 second (base interval of tick). Therefore, one time rule can be evaluated at time X + 5 second, and the next time, for example, Y = X+eval interval, at Y + 2 seconds and therefore the rule is evaluated 3 seconds earlier than it is supposed to.

This is especially important in HA mode where we rely on the fact that rules are evaluated at the same time.

Who is this feature for?
Alerting users

Special notes for your reviewer:

One possible downside of this change is that now all instances in HA mode will send the same query at pretty much the same time, which may cause some contention of the data source side.

Please check that:

  • It works as expected from a user's perspective.
  • If this is a pre-GA feature, it is behind a feature toggle.
  • The docs are updated, and if this is a notable improvement, it's added to our What's New doc.

sort rules scheduled for evaluation to make sure that the order is stable between evaluations.
This is especially important in HA mode.

(cherry picked from commit 05d6813)
@grafana-delivery-bot grafana-delivery-bot bot requested a review from a team as a code owner May 17, 2024 15:41
@grafana-delivery-bot grafana-delivery-bot bot requested review from jtheory, rwwiv, JacobsonMT, yuri-tceretian and grobinson-grafana and removed request for a team May 17, 2024 15:41
@grafana-delivery-bot grafana-delivery-bot bot added this to the 11.0.x milestone May 17, 2024
@yuri-tceretian yuri-tceretian enabled auto-merge (squash) May 20, 2024 13:54
@yuri-tceretian yuri-tceretian merged commit 152cce2 into v11.0.x May 20, 2024
10 checks passed
@yuri-tceretian yuri-tceretian deleted the backport-88006-to-v11.0.x branch May 20, 2024 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

None yet

1 participant