Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cdc: Support schema for Debezium output protocol #10307

Merged
merged 4 commits into from
Dec 27, 2023
Merged

cdc: Support schema for Debezium output protocol #10307

merged 4 commits into from
Dec 27, 2023

Conversation

breezewish
Copy link
Member

@breezewish breezewish commented Dec 14, 2023 •
edited
Loading

What problem does this PR solve?

Issue Number: ref #1799

What is changed and how it works?

  • JSONWriter supports output array
  • Output Debezium schemas
  • Config to enable or disable schema output

Benchmark (with schema):

BenchmarkEncodeOneTinyColumn-10    	  425042	      2734 ns/op	     256 B/op	      12 allocs/op
BenchmarkEncodeLargeText-10        	  342390	      3566 ns/op	     256 B/op	      12 allocs/op
BenchmarkEncodeLargeBinary-10      	  319641	      3497 ns/op	    1410 B/op	      15 allocs/op

Benchmark (without schema):

BenchmarkEncodeOneTinyColumn-10    	 2771671	       430.0 ns/op	       0 B/op	       0 allocs/op
BenchmarkEncodeLargeText-10        	  974575	      1250 ns/op	       0 B/op	       0 allocs/op
BenchmarkEncodeLargeBinary-10      	  978442	      1143 ns/op	    1154 B/op	       3 allocs/op

Check List

Tests

  • Unit test

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with `None`.

@ti-chi-bot ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 14, 2023
@breezewish breezewish mentioned this pull request Dec 14, 2023
Merged
9 tasks
@breezewish
Cleanup history
c56c068 
Signed-off-by: Wish <[email protected]>
@breezewish
Add unit tests
3f5b2cd 
Signed-off-by: Wish <[email protected]>
@sdojjy
Copy link
Member

sdojjy commented Dec 22, 2023

/retest-required

sdojjy
sdojjy reviewed Dec 22, 2023
View reviewed changes
pkg/sink/codec/builder/encoder_builder.go
@@ -49,7 +49,7 @@ func NewRowEventEncoderBuilder(
case config.ProtocolCraft:
return craft.NewBatchEncoderBuilder(cfg), nil
case config.ProtocolDebezium:
return debezium.NewBatchEncoderBuilder(cfg), nil
return debezium.NewBatchEncoderBuilder(cfg, config.GetGlobalServerConfig().ClusterID), nil
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's the ticdc's cluster ID, not the upstream TiDB cluster id

sdojjy
sdojjy reviewed Dec 22, 2023
View reviewed changes
pkg/sink/codec/common/config.go
@@ -138,6 +141,8 @@ type urlConfig struct {
AvroSchemaRegistry string `form:"schema-registry"`
OnlyOutputUpdatedColumns *bool `form:"only-output-updated-columns"`
ContentCompatible *bool `form:"content-compatible"`

DebeziumDisableSchema *bool `form:"debezium-disable-schema"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add this as a part of changefeed toml configuration, not just a sink uri parameter

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added to replica config, PTAL again, thanks!

@sdojjy
Copy link
Member

sdojjy commented Dec 25, 2023

/retest-required

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Dec 25, 2023
@ti-chi-bot ti-chi-bot bot added the lgtm label Dec 26, 2023
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Dec 26, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asddongmen, sdojjy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Dec 26, 2023
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Dec 26, 2023

[LGTM Timeline notifier]

Timeline:

  • 2023-12-25 06:00:37.893426677 +0000 UTC m=+1459128.930653590: ☑️ agreed by sdojjy.
  • 2023-12-26 03:49:23.317880226 +0000 UTC m=+1537654.355107139: ☑️ agreed by asddongmen.

@sdojjy
Copy link
Member

sdojjy commented Dec 27, 2023

/retest-required

1 similar comment
@sdojjy
Copy link
Member

sdojjy commented Dec 27, 2023

/retest-required

@breezewish
Add to config
8b3a1b3 
Signed-off-by: Wish <[email protected]>
@sdojjy
Copy link
Member

sdojjy commented Dec 27, 2023

/retest-required

@ti-chi-bot ti-chi-bot bot merged commit 06c21c8 into pingcap:master Dec 27, 2023
13 checks passed
@breezewish breezewish deleted the d_schema branch December 28, 2023 05:33
@hongyunyan hongyunyan mentioned this pull request Feb 5, 2024
Merged
18 tasks
@qiancai qiancai mentioned this pull request Feb 6, 2024
Merged
17 tasks
@qiancai qiancai mentioned this pull request Feb 6, 2024
Merged
17 tasks
@lilin90 lilin90 mentioned this pull request Mar 7, 2024
Merged
17 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@asddongmen asddongmen asddongmen approved these changes

@sdojjy sdojjy sdojjy approved these changes

@3AceShowHand 3AceShowHand Awaiting requested review from 3AceShowHand

@CharlesCheung96 CharlesCheung96 Awaiting requested review from CharlesCheung96

Assignees
No one assigned
Labels
approved lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@breezewish @sdojjy @asddongmen