How exactly was the bad AM down? Would it have been a failed TCP connection, or would it have hung until timeout?

Whole DC was down, so failed TCP connection

We have a timeout on the sends and do things concurrently, so this doesn't seem likely to be a bug. What I'd say happened is that with so many alerts, you ran into a throughput issue. We do batches of 64 alerts, and there's a default 10s timeout - so that limits things to 6.4 alert/s or around 384 active alerts given the default resend delay is 1m.

So most likely what we need to do here is adjust our defaults to allow for much higher throughput.

Your comment got me thinking & digging further:

The AM instance went down ~ 03:50 UTC on the 24th (unpingable), but the notification queue didn't really build up until the border gateway router went down ~ 11:00.

I'm not certain what the behavior would have been at that point - whether it'd start timing out or just getting ICMP unreachable messages.

Unfortunately for this situation, the border router is back at this point, so I can't test but our legacy Nagios instance was reporting UNREACHABLE for the AM instance at that point (it flipped back to just host down ~ 05:52 the next day).

What does the prometheus_notifications_latency_seconds metric show?

The .5 bucket for that AM instance shot up to 10s as soon as the AM instance went down ~ 04:00. No appreciable change when the AM went unreachable ~ 11:00, but that's when prometheus_notifications_queue_length shot straight up.

Graphs of the above - you can see that latency went up well before the notifications queue filled.

I do have similar issue - 2 DCs, 2xPrometheus + Alertmanagers per DC. DC1 went down for maintenance.

Issue - in DC2, no active alerts in Alertmaanger, while there are in Prometheus ( web ui - Alerts )

From the logs I do see:

Dec 14 17:46:22 s2217.j prometheus[12471]: level=error ts=2023-12-14T17:46:22.713Z caller=notifier.go:527 component=notifier alertmanager=http://s3066.j:9093/api/v2/alerts count=64 msg="Error sending alert" err="Post \"http://s3066.j:9093/api/v2/alerts\": dial tcp 10.215.79.17:9093: i/o timeout"
Dec 14 17:46:32 s2217.j prometheus[12471]: level=warn ts=2023-12-14T17:46:32.238Z caller=notifier.go:379 component=notifier msg="Alert notification queue full, dropping alerts" num_dropped=754
Dec 14 17:46:32 s2217.j prometheus[12471]: level=warn ts=2023-12-14T17:46:32.592Z caller=notifier.go:379 component=notifier msg="Alert notification queue full, dropping alerts" num_dropped=277

From the docs - https://prometheus.io/docs/prometheus/latest/configuration/configuration/#alertmanager_config - I don't see the way to increase the throughtput. The only relevant to my POV setting is in remote_write section, which is for different purposes to my understanding ( https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write )

Prometheus version: prometheus-2.30.0-1.el7.x86_64
Alertmanager version: alertmanager-0.23.0-1.el7.x86_64

Hello from the bug scrub: @krajorama to take an action to verify whether there's a single or multiple queues for the attached AMs in the ruler.

this answers your question, I believe? (i.e. a shared queue, where each batch is sent concurrently to all AMs)

Yes, that certainly looks like it. Also I don't know if the code actually respects the context timeout set on line
544
ctx, cancel := context.WithTimeout(n.ctx, time.Duration(ams.cfg.Timeout))

https://pkg.go.dev/net/http#Request.WithContext

For outgoing client request, the context controls the entire lifetime of a request

on cursory view the timeout(default 10s) should work.

I wrote a unit test in #14099 that shows what I think is happening: alert delivery does not stop , but instead of emptying the queue on each iteration, we incur a 10s penalty (default timeout) for every sendAll call, which means that the throughput drops dramatically. I put some ideas into the PR regarding a solution. Comments welcome on the test as well as the ideas.