Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After the offset is reset and millions of events are fetched and processed again, the memory usage increases sharply when the rd_kafka_commit command is used to submit the offset. The memory usage increases by 4 to 5 GB within 3 to 5 minutes. #4921

Open
3 of 7 tasks
zl2008 opened this issue Dec 3, 2024 · 2 comments

Comments

@zl2008
Copy link

zl2008 commented Dec 3, 2024

Read the FAQ first: https://github.com/confluentinc/librdkafka/wiki/FAQ

Do NOT create issues for questions, use the discussion forum: https://github.com/confluentinc/librdkafka/discussions

Description

After the offset is reset and millions of events are fetched and processed again, the memory usage increases sharply when the rd_kafka_commit command is used to submit the offset. The memory usage increases by 4 to 5 GB within 3 to 5 minutes.

Check logs and find that the connection times out.

log like this:
W001:12-2 00:44:03.389(355899|366121)RDKAFKA-COMMITFAIL: rdkafka#consumer-4: [thrd:main]: Offset commit (manual) failed for 1/1 partition(s) in join-state wait-unassign-to-complete: Local: Waiting for coordinator: ReliableEventPipe[0]@2479111(Local: Waiting for coordinator)
W001:12-2 00:44:03.389(355899|366121)RDKAFKA-COMMITFAIL: rdkafka#consumer-4: [thrd:main]: Offset commit (manual) failed for 1/1 partition(s) in join-state wait-unassign-to-complete: Local: Waiting for coordinator: ReliableEventPipe[0]@2479112(Local: Waiting for coordinator)
W001:12-2 00:44:03.389(355899|366121)RDKAFKA-COMMITFAIL: rdkafka#consumer-4: [thrd:main]: Offset commit (manual) failed for 1/1 partition(s) in join-state wait-unassign-to-complete: Local: Waiting for coordinator: ReliableEventPipe[0]@2479113(Local: Waiting for coordinator)
W001:12-2 00:44:03.389(355899|366121)RDKAFKA-COMMITFAIL: rdkafka#consumer-4: [thrd:main]: Offset commit (manual) failed for 1/1 partition(s) in join-state wait-unassign-to-complete: Local: Waiting for coordinator: ReliableEventPipe[0]@2479114(Local: Waiting for coordinator)

How to reproduce

<your steps how to reproduce goes here, or remove section if not relevant>

IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/confluentinc/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

  • librdkafka version (release number or git tag): <1.9.2>
  • Apache Kafka version: <REPLACE with e.g., 1.9.2>
  • librdkafka client configuration: <REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
  • Operating system: <x86>
  • Provide logs (with debug=.. as necessary) from librdkafka
  • Provide broker log excerpts
  • Critical issue
@zl2008
Copy link
Author

zl2008 commented Dec 3, 2024

version: 1.9.2

@zl2008
Copy link
Author

zl2008 commented Dec 3, 2024

initConsumer([enable.auto.commit, FALSE], [group.id, 165.0.1], [max.poll.interval.ms, 3000000], [queued.max.messages.kbytes, 10000], [queued.min.messages, 10000], [session.timeout.ms, 60000])

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant