Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kv-client(cdc): add more logs for slow regions #10038

Closed
wants to merge 143 commits into from
Closed

kv-client(cdc): add more logs for slow regions #10038

wants to merge 143 commits into from

Conversation

hicqu
Copy link
Contributor

@hicqu hicqu commented Nov 8, 2023

What problem does this PR solve?

Issue Number: ref #9980

What is changed and how it works?

This PR cherry-picks #9981 and some base commits into release-7.1.

Check List

Tests

  • Unit test

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

None

ti-chi-bot added 30 commits April 21, 2023 00:41
@ti-chi-bot
mounter(ticdc): adjust verify checksum method to get the checksum ver…
4d633bf 
…sion (pingcap#8820) (pingcap#8825)

close pingcap#8819
@ti-chi-bot
pkg/tz(ticdc): use correct ctx to get timezone (pingcap#8799) (pingca…
3b9d4d8 
…p#8830)

ref pingcap#8798
@ti-chi-bot
gc (ticdc): optimize the algorithm calculating gc safepoint (pingcap#…
3c6e704 
…8464) (pingcap#8887)

close pingcap#8403
@ti-chi-bot
capture (ticdc): fix processor exit unexpectedly when some pd node fa…
09c43f4 
…il (pingcap#8884) (pingcap#8900)

ref pingcap#8868, close pingcap#8877
@ti-chi-bot
pkg/sink(ticdc): iterate all Kafka configs to support KOP (pingcap#8893…
af0a74c 
…) (pingcap#8906)

close pingcap#8892
@ti-chi-bot
kvclient(cdc): remove rate limiter for scanning and error handling (p…
2ceee3c 
…ingcap#8860) (pingcap#8899)

close pingcap#8858
@ti-chi-bot
Etcd (ticdc, dm):add AutoSyncInterval for Etcd client (pingcap#8813) (p…
b8c9eb7 
…ingcap#8909)

close pingcap#8812
@ti-chi-bot
sink(ticdc): change the directory of storage sink only when ddl event…
705b148 
… occurs (pingcap#8881) (pingcap#8920)

close pingcap#8890, close pingcap#8891
@ti-chi-bot
openprotocol(ticdc): fix open protocol java example (pingcap#8843) (p…
98d0de6 
…ingcap#8851)

close pingcap#8842
@ti-chi-bot
config(ticdc): integrity only works for kafka sink (pingcap#8853) (pi…
265cc9c 
…ngcap#8875)

close pingcap#8854
@ti-chi-bot
sink(ticdc): add index config to storage sink (pingcap#8918) (pingcap…
a5806d9 
…#8929)

close pingcap#8919
@ti-chi-bot
canaljson(ticdc): only output updated column for canal json protocol (p…
ec743d9 
…ingcap#8852) (pingcap#8925)

ref pingcap#8706
@ti-chi-bot
makefile(all): add new cases to run_group and check cases in makefile (…
7080bca 
…pingcap#8794) (pingcap#8924)

close pingcap#8932
@ti-chi-bot
sink(ticdc): move changefeed uri parameter to config file (pingcap#8855…
1512167 
…) (pingcap#8935)

close pingcap#8856
@ti-chi-bot
p2p(ticdc): add keep alive configuration for p2p server (pingcap#8908) (
65306b8 
pingcap#8941)

close pingcap#8907
@ti-chi-bot
pkg/config(ticdc): Support InsecureSkipVerify option when using tls (p…
2c3b177 
…ingcap#8873) (pingcap#8937)

close pingcap#8867
@ti-chi-bot
p2p(ticdc): remove message peer when capture is removed (pingcap#8923) (
235705e 
pingcap#8943)

close pingcap#8922
@ti-chi-bot
api(ticdc): add changefeed status api (pingcap#8872) (pingcap#8945)
24abb01 
close pingcap#8879
@ti-chi-bot
integrity(ticdc): add integrity package and move some files, reorder …
672e2e9 
…columns at the mounter (pingcap#8917) (pingcap#8942)

close pingcap#8916
@ti-chi-bot
sink(ticdc): add meta separator index path in storage sink (pingcap#8948
6afe6df 
) (pingcap#8952)

ref pingcap#8890
@ti-chi-bot
schema, sink(ticdc): fix exchange partition (pingcap#8955) (pingcap#8962
ca9c4e2 
)

close pingcap#8914
@ti-chi-bot
mounter(ticdc): fix extend column info order to match the columns info (
d8cae62 
pingcap#8960) (pingcap#8969)

close pingcap#8958
@ti-chi-bot
metrics (ticdc): remove p99 lag distribution (pingcap#8967) (pingcap#…
91f9508 
…8979)

ref pingcap#8871, close pingcap#8971
@ti-chi-bot
sink(cdc): handle sink errors more fast and light (pingcap#8949) (pin…
f2cd956 
…gcap#8976)

ref pingcap#8657
@ti-chi-bot
doc(ticdc): update design doc of storage sink (pingcap#8954) (pingcap…
c96511c 
…#8978)

ref pingcap#8890
@ti-chi-bot
makefile, cmd(ticdc): log version in consumer (pingcap#8981) (pingcap…
265ddde 
…#8988)

ref pingcap#8973
@ti-chi-bot
(sink/cdc): fix stop table dead lock introduced in pingcap#8949 (ping…
2ea7904 
…cap#8989) (pingcap#8994)

ref pingcap#8657
@ti-chi-bot
owner, redo(ticdc): decouple globalResolvedTs and globalBarrierTs (pi…
d877c76 
…ngcap#8964) (pingcap#8996)

close pingcap#8963
@ti-chi-bot
tests(ticdc): add lossy_ddl test (pingcap#8859) (pingcap#8985)
29afa87 
close pingcap#8686
@ti-chi-bot
(processor/cdc): report module internal warnings (pingcap#8983) (ping…
04c2964 
…cap#9002)

ref pingcap#8657
ti-chi-bot and others added 22 commits October 7, 2023 10:19
@ti-chi-bot
config(ticdc): mask more security query parameter (pingcap#9832) (pin…
a66f484 
…gcap#9836)

close pingcap#9831
@CharlesCheung96
scheduler (ticdc): does not return error when resolvedTs less than ch…
2449b3b 
…eckpoint (pingcap#9833)

ref pingcap#9830
@ti-chi-bot
scheduler(ticdc): fix incorrect scheduling task counter (pingcap#9840) (
361f541 
pingcap#9847)

close pingcap#9845
@ti-chi-bot
sink(cdc): fix the bug that mq sink can lost callbacks (pingcap#9852) (…
13de612 
…pingcap#9859)

close pingcap#9855
@ti-chi-bot
ddlManager (ticdc): Fix a bug that could lead to TiCDC replicating DD…
3777731 
…L events of ineligible tables. (pingcap#9846) (pingcap#9854)

close pingcap#9819
@3AceShowHand
kafka(ticdc): fix kafka dml sink report error to sink manager (pingca…
07ad2c1 
…p#9879)

ref pingcap#9855
@ti-chi-bot
ddl(ticdc): async exec add index ddl (pingcap#9701) (pingcap#9874)
f3cf834 
close pingcap#9644
@ti-chi-bot
sorter(cdc): fix panics of accessing sort engine (pingcap#9882) (ping…
b2d0d7b 
…cap#9889)

close pingcap#9849
@ti-chi-bot @GMHDBJD
reset optimistic sharding group keeper when syncer resume (pingcap#9720)
380b549 
Co-authored-by: gmhdbjd <[email protected]>
@ti-chi-bot @GMHDBJD
dm: track table schema for online ddl when use binlog skip (pingcap#9700
dd7ccf4 
) (pingcap#9706)

* apply online ddl when handle-error skip

* update

---------

Co-authored-by: gmhdbjd <[email protected]>
@ti-chi-bot
config, changefeed (ticdc): add changefeed error stuck duration config (
172a8e7 
pingcap#9872) (pingcap#9878)

close pingcap#9875
@ti-chi-bot
dm: sync partition DDL even if upstream schema is not equal in optimi…
69e9d10 
…stic mode (pingcap#9789) (pingcap#9805)
@ti-chi-bot
dm: add strict optimist mode (pingcap#9113) (pingcap#9540)
ff869a6 
close pingcap#9112
@ti-chi-bot
scheduler(ticdc): fix invlaid checkpoint when redo enabled (pingcap#9851
bef3924 
) (pingcap#9907)

close pingcap#9774, close pingcap#9830
@ti-chi-bot
reset internal state of feedStateManager when shouldBeRunning is false (
64790ca 
pingcap#9893) (pingcap#9905)

close pingcap#9892
@ti-chi-bot
changefeed(ticdc): check changefeed info is nil to prevent panic (pin…
05f70a9 
…gcap#9917) (pingcap#9923)

close pingcap#9915
@CharlesCheung96
Revert bef3924 in release 7.1 (pingcap#9932)
6ba0aec 
ref pingcap#9926
@ti-chi-bot
mysql(ticdc): use string join to build multi statement sql. (pingcap#…
4439ae4 
…9936) (pingcap#9937)

close pingcap#9935
@ti-chi-bot
kafka(ticdc): topic manager return the partition number specified in …
6eaec3d 
…the sink-uri (pingcap#9955) (pingcap#9965)

close pingcap#9952
@ti-chi-bot
redo(ticdc): fix redo initialization block the owner (pingcap#9887) (p…
ba5bced 
…ingcap#9992)

close pingcap#9886
@hicqu
init
06c6ae2 
Signed-off-by: qupeng <[email protected]>
@hicqu
pick more to 7.1
9216167 
Signed-off-by: qupeng <[email protected]>
@ti-chi-bot ti-chi-bot bot added release-note-none Denotes a PR that doesn't merit a release note. needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. labels Nov 8, 2023
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Nov 8, 2023

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from hicqu. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 8, 2023
@hicqu hicqu closed this Nov 8, 2023
@hicqu hicqu deleted the release-7.1-client-logs branch November 8, 2023 07:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@hicqu @ti-chi-bot @CharlesCheung96 @asddongmen @3AceShowHand