VMAgent OOMKilled after upgrading to v1.97.1

[just1900]

Is your question request related to a specific component?

vmagent

Describe the question in detail

I find VMAgent got OOMKilled after upgrading from v1.79 -> v1.97
and its memory usage fluctuate significantly, what might be the cause?

Troubleshooting docs

• General - https://docs.victoriametrics.com/troubleshooting/
• vmagent - https://docs.victoriametrics.com/vmagent/#troubleshooting
• vmalert - https://docs.victoriametrics.com/vmalert/#troubleshooting

[f41gh7]

Hello, could you please collect and attach memory profile for vmagent in the moment of high memory usage?

Most probably, it's related to the size of targets scrape response.

https://docs.victoriametrics.com/#profiling

[just1900]

It seems that the application's qps is often higher than usual, this might result in generating larger responses for /metrics, when vmagent has high memory usage.

I will share the profile here if it happens again.

[antoinedeschenes]

Not sure I should create a separate thread, but we're seeing that 6 of ~30 vmagents have seen the memory usage double when upgrading from v1.93.14 to v1.101.0, running the same chart and operator versions. No difference in target count.
pprof.zip