Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove Kinesis producer's internal TTL by default #2147

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

msg555
Copy link
Contributor

@msg555 msg555 commented Jan 3, 2025

By default the Kinesis producer implementation will time out records that cannot be delivered within 30 seconds. This interferes with Maxwell's back-pressure implementation as more than 30 seconds of records may buffer before Maxwell begins to throttle itself causing the whole daemon to exit.

I initially tried setting the Ttl to the max value mentioned in the Kinesis producer documentation, unfortunately this isn't actually supported by the Kinesis producer since it appears to cause overflow in the backend and makes the records timeout immediately. Opted for just changing the Ttl to 1 hour instead which I imagine is a reasonable compromise that let's Maxwell's back pressure do its thing while still surfacing an error if records long term continue to be undeliverable.

Tested the change and can confirm I'm not seeing failures in the timeout failures anymore / back pressure is working properly.

Reference

See https://javadoc.io/doc/com.amazonaws/amazon-kinesis-producer/0.14.0/com/amazonaws/services/kinesis/producer/KinesisProducerConfiguration.html#getRecordTtl--

@msg555 msg555 changed the title Remove Kinesis producer's internal TTL by default WIP: Remove Kinesis producer's internal TTL by default Jan 3, 2025
@osheroff
Copy link
Collaborator

osheroff commented Jan 3, 2025

fine by me, lmk when you want it merged

@msg555 msg555 force-pushed the msg/remove-kinesis-ttl branch from b267fd1 to cac143a Compare January 3, 2025 18:23
@msg555 msg555 force-pushed the msg/remove-kinesis-ttl branch from cac143a to b98b036 Compare January 3, 2025 18:29
@msg555 msg555 changed the title WIP: Remove Kinesis producer's internal TTL by default Remove Kinesis producer's internal TTL by default Jan 3, 2025
@msg555
Copy link
Contributor Author

msg555 commented Jan 3, 2025

Should be ready to go. Happy to make any changes you want though.

I've resolved my issue by passing a custom kinesis producer configuration now, so no urgency on my end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants