You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After the offset is reset and millions of events are fetched and processed again, the memory usage increases sharply when the rd_kafka_commit command is used to submit the offset. The memory usage increases by 4 to 5 GB within 3 to 5 minutes.
#4921
After the offset is reset and millions of events are fetched and processed again, the memory usage increases sharply when the rd_kafka_commit command is used to submit the offset. The memory usage increases by 4 to 5 GB within 3 to 5 minutes.
Check logs and find that the connection times out.
log like this:
W001:12-2 00:44:03.389(355899|366121)RDKAFKA-COMMITFAIL: rdkafka#consumer-4: [thrd:main]: Offset commit (manual) failed for 1/1 partition(s) in join-state wait-unassign-to-complete: Local: Waiting for coordinator: ReliableEventPipe[0]@2479111(Local: Waiting for coordinator)
W001:12-2 00:44:03.389(355899|366121)RDKAFKA-COMMITFAIL: rdkafka#consumer-4: [thrd:main]: Offset commit (manual) failed for 1/1 partition(s) in join-state wait-unassign-to-complete: Local: Waiting for coordinator: ReliableEventPipe[0]@2479112(Local: Waiting for coordinator)
W001:12-2 00:44:03.389(355899|366121)RDKAFKA-COMMITFAIL: rdkafka#consumer-4: [thrd:main]: Offset commit (manual) failed for 1/1 partition(s) in join-state wait-unassign-to-complete: Local: Waiting for coordinator: ReliableEventPipe[0]@2479113(Local: Waiting for coordinator)
W001:12-2 00:44:03.389(355899|366121)RDKAFKA-COMMITFAIL: rdkafka#consumer-4: [thrd:main]: Offset commit (manual) failed for 1/1 partition(s) in join-state wait-unassign-to-complete: Local: Waiting for coordinator: ReliableEventPipe[0]@2479114(Local: Waiting for coordinator)
How to reproduce
<your steps how to reproduce goes here, or remove section if not relevant>
IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/confluentinc/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.
Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
librdkafka version (release number or git tag): <1.9.2>
Apache Kafka version: <REPLACE with e.g., 1.9.2>
librdkafka client configuration: <REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
Operating system: <x86>
Provide logs (with debug=.. as necessary) from librdkafka
Provide broker log excerpts
Critical issue
The text was updated successfully, but these errors were encountered:
Read the FAQ first: https://github.com/confluentinc/librdkafka/wiki/FAQ
Do NOT create issues for questions, use the discussion forum: https://github.com/confluentinc/librdkafka/discussions
Description
After the offset is reset and millions of events are fetched and processed again, the memory usage increases sharply when the rd_kafka_commit command is used to submit the offset. The memory usage increases by 4 to 5 GB within 3 to 5 minutes.
Check logs and find that the connection times out.
log like this:
W001:12-2 00:44:03.389(355899|366121)RDKAFKA-COMMITFAIL: rdkafka#consumer-4: [thrd:main]: Offset commit (manual) failed for 1/1 partition(s) in join-state wait-unassign-to-complete: Local: Waiting for coordinator: ReliableEventPipe[0]@2479111(Local: Waiting for coordinator)
W001:12-2 00:44:03.389(355899|366121)RDKAFKA-COMMITFAIL: rdkafka#consumer-4: [thrd:main]: Offset commit (manual) failed for 1/1 partition(s) in join-state wait-unassign-to-complete: Local: Waiting for coordinator: ReliableEventPipe[0]@2479112(Local: Waiting for coordinator)
W001:12-2 00:44:03.389(355899|366121)RDKAFKA-COMMITFAIL: rdkafka#consumer-4: [thrd:main]: Offset commit (manual) failed for 1/1 partition(s) in join-state wait-unassign-to-complete: Local: Waiting for coordinator: ReliableEventPipe[0]@2479113(Local: Waiting for coordinator)
W001:12-2 00:44:03.389(355899|366121)RDKAFKA-COMMITFAIL: rdkafka#consumer-4: [thrd:main]: Offset commit (manual) failed for 1/1 partition(s) in join-state wait-unassign-to-complete: Local: Waiting for coordinator: ReliableEventPipe[0]@2479114(Local: Waiting for coordinator)
How to reproduce
<your steps how to reproduce goes here, or remove section if not relevant>
IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/confluentinc/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.
Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
<1.9.2>
<REPLACE with e.g., 1.9.2>
<REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
<x86>
debug=..
as necessary) from librdkafkaThe text was updated successfully, but these errors were encountered: