feat(processors): Traffic shaper processor plugin to shape uneven distribution of incoming metrics #15353
Labels
feature request
Requests for new plugin and for new features to existing plugins
Comments
lakshmansai
added
the
feature request
|
Hi, After discussing with the team, we would prefer that rate limiting like this occur in outputs instead of a processor. We have an open issue for this in: #14802 That is where I would want to continue this discussion. The processor solution unfortunately adds a second buffer, which introduces another set of issues. You can see some examples of this #802 for a way to maintain the buffer outside of memory via a WAL and how to control the buffer itself with #15208 and #15217. As such we do not wish to add a processor, but would take a PR to look at output rate limiting. I will close this issue and the PR. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Use Case
An in-memory traffic shaper processor which evens out traffic so that output traffic is uniform
We use telegraf as a proxy and we receive data that is spiky in nature, in our every 10 minute we receive a spike and this affects our downstream systems since it needs to also process at the same rate, this leads to wastage of resource since the cpu, memory needs to be provisioned for peaks
Screenshot showing spiky nature of data
Expected behavior
Actual behavior
We have implemented a traffic shaper plugin in telegraf which evens out the spike and help us maintain a steady rate.
Example post using traffic shaper plugin we can see after 00:00 the traffic became uniform
Additional info
No response
The text was updated successfully, but these errors were encountered: