Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add conf to choose if appending thread number to client_id or not #54

Open
arman1371 opened this issue Aug 28, 2020 · 1 comment · May be fixed by #55
Open

Add conf to choose if appending thread number to client_id or not #54

arman1371 opened this issue Aug 28, 2020 · 1 comment · May be fixed by #55

Comments

@arman1371
Copy link

  • Version: 10.5.1

Sometimes users uses kafka client_id to set quotas for input/output IO of kafka consumers/producers. so just some specific client ids can read/write from kafka cluster with a high rate.
Now logstash-integration-kafka plugin appends the thread-id to client_id without any documentation and there isn't any config to disable it to just send the client_id to kafka cluster.

@kares
Copy link
Contributor

kares commented Oct 21, 2020

Thanks Arman, this seems like a legit use-case.
Believe the plugin should not change client_id esp. if it's set to an explicit value by the user.

Now, I am not 100% how to approach this one but #55 seems a bit 'ugly' with an option to change another option.
Maybe we can do some kind of formatting on the client_id thus respecting an explicitly set value while also allowing to use the current behavior (not changing current behavior when client_id is not set - should be the common case) e.g. :

config :client_id, :validate => :string, :default => "logstash-%i"

and then sprintf-ing client_id against the thread counter.
It might still needs more tuning as this might seem too similar to LS' sprintf-ing of event fields (using the %{} syntax) and thus might lead to a bit of confusion.

A reminder on how Kafka uses client_id ...

sole purpose of this is to be able to track the source of requests beyond just ip and port by allowing a logical application name to be included in Kafka logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants