Add metrics generation for on-call schedule

[shurkys]

What this PR does

This PR adds functionality to generate metrics for on-call schedules. It introduces a method _get_schedule that retrieves on-call schedules for specified organization IDs, iterates through each schedule, and collects metrics for each user and team combination in the schedule. This enhancement improves the monitoring and analysis capabilities of the application by providing insights into the on-call rotation and team responsibilities.

Output example:
oncall_schedule{schedule="Schedule1",team="test_team1",user="oncall"} 1.0

Which issue(s) this PR fixes

Closes #3427

Checklist

• Unit, integration, and e2e (if applicable) tests updated
• Documentation added (or pr:no public docs PR label added if not required)

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ joeyorlando
❌ shurkys

---

shurkys seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[joeyorlando]

thanks for the contribution @shurkys! Do you mind adding a small section about the new metric to the docs here. Thank you! 🙏

[shurkys]

Added documentation and renamed the user label to the username for better readability

[matiasb]

Nice! This seems a good idea 👍 , adding some suggestion. It would be also great to add a few tests.

[matiasb]

I think this loop may be too expensive to run for multiple orgs/schedules (and make the scrape timeout). Have you considered using a similar approach to the other metrics keeping data in cache and just fetching from there when running the collector? Note that we already have a periodical task checking who is on-call for each schedule (refresh_ical_file) which already updates cache (update_cached_oncall_users_for_schedule). I think it should be possible to plug this there?

[shurkys]

Updated. I don’t know how to optimize it even more.

[matiasb]

FYI, schedule.team may be None, so you would need something like:
schedule.team.name if schedule.team else "No team"
(to match the value used in the other metrics linking to teams)

[shurkys]

Good point. Corrected

[matiasb]

This is a good update to make things more efficient, but this could still trigger requests to fetch imported ical files from ical-based schedules, making the method too slow in this case.

I think the best path forward for this would be to implement the logic here using cached data only, which potentially means reworking what data we cache for schedules in the refresh task (ie. the data you use here) and implement some additional flexibility to only use information we have in cache (otherwise fetching all schedules from DB and iterate on them, potentially triggering external http requests, and later also getting user details from DB, will make the exporter API timeout).

Hope that makes sense, I can take a look and try to draft something if I get some time available.

[shurkys]

Revised the code. API calls also use queryset = OnCallSchedule.objects.all(). There was no caching in the schedulers itself and OnCallSchedule.objects.all().query says that a query is constantly being generated into the database.
Now /api/v1/schedules/ is in use and does not cause any problems with load or delay in calls. I don’t see any restrictions why the same approach cannot be used now for metrics, since there is no other way.
As for optimization, it seems the focus should be on improving the scheduler itself and implementing caching mechanisms. I looked in there, got scared, and closed it.

[matiasb]

Hm.. I see, that queryset definition there shouldn't be needed and I think it is not being used (note the API class defines and uses the get_queryset method, where the query is limited per org).
In any case, the main issue here is that for schedules not in cache, the get_cached_oncall_users_for_multiple_schedules may trigger http requests to refresh imported ical files, making this exporter method take too long and then the prometheus scrape would timeout (FWIW, I confirmed this manually running the logic in our stack).
I agree this could work ok in a local setup, but we would need some additional work and optimization before merging and enabling it in cloud (which as I said I would be interested to enable, so I will take a look when I get some time). Makes sense?

[shurkys]

Understood, it seems like it indeed makes sense.

[github-actions]

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in 30 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!