Kafka Input plugin - allow access to broker list through a VIP

[nbcohen]

Use Case

Current format in kafka input plugin:

[[inputs.kafka_consumer]]
brokers = [ "myBroker1:1234, myBroker2:1234" ]
topics = [ "myopic" ]
precision = "1s"
etc. etc.

The brokers entry tells me which kafka broker to connect to.

Our kafka team has set up a VIP entry which maps to a list of a dozen or more brokers, so if one is down, the connection should still work.

The Sarama library supports VIP access in telegraf 1.29+ and we are using telegraf 1.30 at the moment so we should have the capability.

There is a new parameter - client.dns.lookup: resolve_canonical_bootstrap_servers_only

which seems to be required for this to work, but telegraf doesn't currently support it. Can you add the capability to use that Sarama feature so I can set brokers="myVIP:1234" and have it resolve to the broker list...

Thanks,

nbc

Expected behavior

With this new parameter we should be able to list only the VIP address in the telegraf config and have it connect to any available kafka broker.

Actual behavior

Not currently available

Additional info

No response

[InfluxJay]

Potentially, this would allow the Kafka admins to take brokers out of service for maintenance without affecting the telegraf processing, which I believe is the aim @nbcohen ?

[powersj]

@nbcohen,

I've put up #15368 which will have artifacts attached in a few moments. Please download and try the artifacts and let us know if exposing the new config option resolves your request.

In your config you will need to add the following to inputs.kafka_consumer`:

[[inputs.kafka_consumer]]
  resolve_canonical_bootstrap_servers_only = true

Thanks!

[nbcohen]

Hi - Thanks very much for the quick turnaround... It may take me longer to test it than for you to fix it... I had to put in the request from my personal account - could not do it from work. I will now need to get permission to install your update at the office. We have an influx support contract, so I hope it won't be a problem for me. I will let you know as soon as I can get it tested. Probably will be some time next week. nbc

…

On Thu, May 16, 2024 at 10:08 AM Joshua Powers ***@***.***> wrote: @nbcohen <https://github.com/nbcohen>, I've put up #15368 <#15368> which will have artifacts attached in a few moments. Please download and try the artifacts and let us know if exposing the new config option resolves your request. In your config you will need to add the following to inputs.kafka_consumer`: [[inputs.kafka_consumer]] resolve_canonical_bootstrap_servers_only = true Thanks! — Reply to this email directly, view it on GitHub <#15367 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA22I6XPALE7UZKTVXDGXIDZCS4UHAVCNFSM6AAAAABH2C6MN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJVGM2TCNJUGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- NAME: Neil B. Cohen (Aiki Software) PHONE: 703-444-4610 DOMAIN: ***@***.*** ************************************************************* * Murphy's Philosophy: Smile - tomorrow will be worse... * * O'Tooles Commentary: Murphy was an optimist! *************************************************************

[powersj]

Hi,

I'm going to have the team review the PR and probably land this. Our next release is coming up and I don't want to miss this.

Thanks!

[nbcohen]

thanks! the release will probably come out before I get permission to install your test version. I look forward to using it NBC

…

On Fri, May 31, 2024, 4:28 AM Sven Rebhan ***@***.***> wrote: Closed #15367 <#15367> as completed via #15368 <#15368>. — Reply to this email directly, view it on GitHub <#15367 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA22I6RCAAFBHEABXNZYUGTZFAYBTAVCNFSM6AAAAABH2C6MN2VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJSHE4TMNRYGU3DOMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>