Add CompressionLevel and make v2 Kafka sink default #19169
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Files changed: |
✅ Deploy Preview for cockroachdb-api-docs canceled.
|
✅ Deploy Preview for cockroachdb-interactivetutorials-docs canceled.
|
✅ Netlify Preview
To edit notification comments on pull requests, go to your Netlify site configuration. |
kathancox
force-pushed
the
kafka-v2-sink-updates
branch
from
53b6051
to
c9da936
Compare
kathancox
force-pushed
the
kafka-v2-sink-updates
branch
from
c9da936
to
de9c4ec
Compare
kathancox
commented
Nov 21, 2024
| The `kafka_sink_config` option allows configuration of a changefeed's message delivery, Kafka server version, and batching parameters. | ||
| You can configure flushing, acknowledgments, compression, and concurrency behavior of changefeeds running to a Kafka sink with the following: | ||
|
|
||
| - Set the [`changefeed.sink_io_workers` cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}#setting-changefeed-sink-io-workers) to configure the number of concurrent workers used by changefeeds in the cluster when sending requests to a Kafka sink. When you set `changefeed.sink_io_workers`, it will not affect running changefeeds; [pause the changefeed]({% link {{ page.version.version }}/pause-job.md %}), set `changefeed.sink_io_workers`, and then [resume the changefeed]({% link {{ page.version.version }}/resume-job.md %}). Note that this cluster setting will also affect changefeeds running to [Google Cloud Pub/Sub](#google-cloud-pub-sub) sinks and [webhook sinks](#webhook-sink). |
There was a problem hiding this comment.
Does it make sense to add this now that the v2 Kafka sink is the default? (This paragraph is included for Pub/Sub + Webhook too.)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes DOC-11339, DOC-10867, DOC-10830, DOC-10700
This PR:
kafka_sink_configoption with theCompressionLevelfield in v24.3.changefeed.sink-io-workersunder Kafka for the default v2 sink.Rendered Preview
https://deploy-preview-19169--cockroachdb-docs.netlify.app/docs/v24.3/changefeed-sinks.html#kafka-sink-configuration