Possible inconsistency in configuration parameter doc

[FFrederic]

Hello,

I was trying to tune the kafka consumer configuration parameters listed in docs/include/plugin_header-integration.asciidoc

If I am reading it right, there seems to be an inconsistency between the default values of two configuration parameters and the suggestion that one should be lower than the other.

===== fetch_max_wait_ms

• Default value is 500 milliseconds.

This should be less than or equal to the timeout used in poll_timeout_ms

...

===== poll_timeout_ms

• Default value is 100 milliseconds.

[jsvd]

The recommendation does makes sense IMO. Having poll timeouts higher than fetch.max.wait.ms means that each poll request lets the kafka broker respond whenever it reaches fetch.min.bytes of data: either immediately or up to fetch.max.wait.ms.
The poll should never reach the timeout, we only need it if the kafka server doesn't respond in time for some reason and we want to terminate the pipeline, so we need the timeout to unblock the plugin thread.

That said, most users shouldn't need to touch either of these unless they have very low EPS but need the data with very low latency. In this case lowering fetch.max.wait.ms and perhaps even fetch.min.bytes will help.

From my understanding I think the best course of action here would be to set the default poll timeout to 1000ms to avoid having polls timeout, but as mentioned above, no ill should come from the current defaults.