Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sink(ticdc): use multi part s3 uploader in storage sink #9954

Merged
merged 2 commits into from
Nov 29, 2023
Merged

sink(ticdc): use multi part s3 uploader in storage sink #9954

merged 2 commits into from
Nov 29, 2023

Conversation

CharlesCheung96
Copy link
Contributor

@CharlesCheung96 CharlesCheung96 commented Oct 25, 2023 •
edited
Loading

What problem does this PR solve?

Issue Number: close #10098, close #10172

What is changed and how it works?

Add support for multipart upload to Amazon S3 in the storage sink.

Check List

Tests

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note 

`Add support for multipart upload to Amazon S3 in the storage sink.`.

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Oct 25, 2023

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Oct 25, 2023
@CharlesCheung96 CharlesCheung96 mentioned this pull request Nov 2, 2023
Merged
@CharlesCheung96 CharlesCheung96 changed the title sink(ticdc): use multi part uploader in storage sink sink(ticdc): use multi part s3 uploader in storage sink Nov 9, 2023
@CharlesCheung96 CharlesCheung96 marked this pull request as ready for review November 10, 2023 02:26
@ti-chi-bot ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 10, 2023
@CharlesCheung96 CharlesCheung96 mentioned this pull request Nov 13, 2023
Merged
@CharlesCheung96 CharlesCheung96 force-pushed the fix-storage-single-table branch from 4362870 to 487cd7d Compare November 15, 2023 03:18
@CharlesCheung96
Copy link
Contributor Author

/retest

@ti-chi-bot ti-chi-bot bot added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. and removed do-not-merge/needs-linked-issue labels Nov 15, 2023
@CharlesCheung96
Copy link
Contributor Author

/retest

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Nov 16, 2023
@ti-chi-bot ti-chi-bot mentioned this pull request Nov 20, 2023
Merged
@ti-chi-bot ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 23, 2023
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Nov 23, 2023

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ti-chi-bot ti-chi-bot bot added the lgtm label Nov 27, 2023
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Nov 27, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nongfushanquan, sdojjy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot removed the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Nov 27, 2023
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Nov 27, 2023

[LGTM Timeline notifier]

Timeline:

  • 2023-11-16 05:37:33.110155802 +0000 UTC m=+4313850.697265946: ☑️ agreed by sdojjy.
  • 2023-11-27 00:31:08.872045581 +0000 UTC m=+796297.537271777: ☑️ agreed by nongfushanquan.

@CharlesCheung96 CharlesCheung96 force-pushed the fix-storage-single-table branch from 487cd7d to 7902936 Compare November 28, 2023 11:55
@CharlesCheung96 CharlesCheung96 removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 28, 2023
@CharlesCheung96
Copy link
Contributor Author

/retest

@CharlesCheung96
Copy link
Contributor Author

/check-issue-triage-complete

@CharlesCheung96
Copy link
Contributor Author

/retest

2 similar comments
@CharlesCheung96
Copy link
Contributor Author

/retest

@CharlesCheung96
Copy link
Contributor Author

/retest

@ti-chi-bot ti-chi-bot bot merged commit b6c1a94 into pingcap:master Nov 29, 2023
14 checks passed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #10178.

@ti-chi-bot ti-chi-bot mentioned this pull request Nov 29, 2023
Merged
ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 29, 2023
@CharlesCheung96 @ti-chi-bot
This is an automated cherry-pick of pingcap#9954
c2f423f 
Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.1: #10179.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 29, 2023
@CharlesCheung96 @ti-chi-bot
This is an automated cherry-pick of pingcap#9954
75a3c9c 
Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this pull request Nov 29, 2023
Merged
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.5: #10180.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 29, 2023
@CharlesCheung96 @ti-chi-bot
This is an automated cherry-pick of pingcap#9954
54b1ffc 
Signed-off-by: ti-chi-bot <[email protected]>
@ti-chi-bot ti-chi-bot mentioned this pull request Nov 29, 2023
Merged
3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this pull request Nov 29, 2023
@CharlesCheung96 @3AceShowHand
sink(ticdc): use multi part s3 uploader in storage sink (pingcap#9954)
a5642d2 
close pingcap#10098, close pingcap#10172
ti-chi-bot bot pushed a commit that referenced this pull request Nov 30, 2023
@ti-chi-bot
sink(ticdc): use multi part s3 uploader in storage sink (#9954) (#10178)
873a2ff 
close #10098, close #10172
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request Dec 7, 2023
@CharlesCheung96
sink(ticdc): use multi part s3 uploader in storage sink (pingcap#9954)
f2070f6 
close pingcap#10098, close pingcap#10172
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request Dec 8, 2023
@CharlesCheung96
sink(ticdc): use multi part s3 uploader in storage sink (pingcap#9954)
8cb3218 
close pingcap#10098, close pingcap#10172
ti-chi-bot bot pushed a commit that referenced this pull request Dec 8, 2023
@ti-chi-bot
sink(ticdc): use multi part s3 uploader in storage sink (#9954) (#10179)
7099e8b 
close #10098, close #10172
@D3Hunter D3Hunter mentioned this pull request Dec 11, 2023
Closed
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request Feb 9, 2024
@CharlesCheung96
sink(ticdc): use multi part s3 uploader in storage sink (pingcap#9954)
1ce569c 
close pingcap#10098, close pingcap#10172
CharlesCheung96 added a commit to ti-chi-bot/tiflow that referenced this pull request Feb 9, 2024
@CharlesCheung96
sink(ticdc): use multi part s3 uploader in storage sink (pingcap#9954)
4d7324d 
close pingcap#10098, close pingcap#10172
ti-chi-bot bot pushed a commit that referenced this pull request Feb 9, 2024
@ti-chi-bot
sink(ticdc): use multi part s3 uploader in storage sink (#9954) (#10180)
df3ee6a 
close #10098, close #10172
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhangjinpeng87 zhangjinpeng87 zhangjinpeng87 left review comments

@nongfushanquan nongfushanquan nongfushanquan approved these changes

@sdojjy sdojjy sdojjy approved these changes

Assignees
No one assigned
Labels
approved lgtm needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

unstable unit test TestGCAndCleanup use multi part s3 uploader in storage sink
5 participants
@CharlesCheung96 @ti-chi-bot @sdojjy @zhangjinpeng87 @nongfushanquan