"Local: Message timed out" when publishing message to Kafka

[justrebeka]

Hello, we are using Confluent Kafka in a .net service and sometimes our service is encountering issues with publishing messages to Kafka. It seems message.timeout.ms is exceeded and an error is returned form librdkafka: Local: message timed out. As far as I have read, this is an internal error,:

The message could not be successfully transmitted before message.timeout.ms expired, typically due to no leader being available or no broker connection. The message may have been retried due to other errors but those error messages are abstracted by the ERR__MSG_TIMED_OUT error code.

after enabling the logs, i noticed this in the logs:

Broker changed state CONNECT -> DOWN
2024-11-27 01:00:10.706 +00:00 [DBG] Kafka processor error log: IsFatal:False IsLocalError:True IsBrokerError:FalseCode:sasl_ssl://b-3.mskmessagingcluster.abc123.c2.kafka.us-east-2.amazonaws.com:9098/3: Connect to ipv4#10.0.121.123:9098 failed: Unknown error (after 21025ms in state CONNECT, 2 identical error(s) suppressed) Reason:sasl_ssl://b-3.mskmessagingcluster.abc123.c2.kafka.us-east-2.amazonaws.com:9098/3: Connect to ipv4#10.0.121.123:9098 failed: Unknown error (after 21025ms in state CONNECT, 2 identical error(s) suppressed)

Please note that I didn't change the default configuration for message.timeout.ms. However, after a restart our service can publish successfully data for some time.

I checked KafkaDataLogsDiskUsed metric but I didn't find this going more than 85%. however I can see the trend is for it to increase:

.
Any thoughts about how I could continue with the troubleshooting? thank you?