Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sink(ticdc): cleanup expired files by day for storage sink #10097

Merged
merged 10 commits into from
Nov 26, 2023

Conversation

CharlesCheung96
Copy link
Contributor

@CharlesCheung96 CharlesCheung96 commented Nov 14, 2023

What problem does this PR solve?

Issue Number: close #10109

What is changed and how it works?

Add two parameters to control the cleaning behavior of expired files, for example:

  1. To keep one day's files and clean them at 18 o 'clock every day, users can use the following configuration.
[sink.cloud-storage-config]
file-expiration-days = 1
# Second | Minute | Hour | Dom | Month | DowOptional
file-cleanup-cron-spec = "0 18 * * * *"    
  1. To keep files for a week and clean them every Monday at 2 am, you can use the following configuration.
[sink.cloud-storage-config]
file-expiration-days = 7
# Second | Minute | Hour | Dom | Month | DowOptional
file-cleanup-cron-spec = "0 2 * * * 1"
  1. The default configuration is
[sink.cloud-storage-config]
file-expiration-days = 0 # The default congiuration
file-cleanup-cron-spec = "0 2 * * * *"
  • By default, files are not cleaned.
  • If only the file-expiration-days is configured, then by default cdc will clean files at 2 a.m. each day.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

`Add file-expiration-days and file-cleanup-cron-spec configurations for storage sink to control the cleaning behavior of expired files`.

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Nov 14, 2023
Copy link
Contributor

ti-chi-bot bot commented Nov 14, 2023

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 14, 2023
Copy link
Contributor

@nongfushanquan nongfushanquan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add the user manual in the pr content ,such as some limitations

@ti-chi-bot ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. do-not-merge/needs-triage-completed and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. do-not-merge/needs-linked-issue labels Nov 16, 2023
@CharlesCheung96
Copy link
Contributor Author

/test all

@CharlesCheung96 CharlesCheung96 marked this pull request as ready for review November 20, 2023 02:47
@ti-chi-bot ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 20, 2023
@CharlesCheung96
Copy link
Contributor Author

/check-issue-triage-complete

@ti-chi-bot ti-chi-bot bot added needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed do-not-merge/needs-triage-completed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 20, 2023
@ti-chi-bot ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Nov 22, 2023
@ti-chi-bot ti-chi-bot bot added approved and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Nov 23, 2023
Copy link
Contributor

ti-chi-bot bot commented Nov 23, 2023

[LGTM Timeline notifier]

Timeline:

  • 2023-11-22 05:58:43.461748959 +0000 UTC m=+383952.126975152: ☑️ agreed by nongfushanquan.
  • 2023-11-23 06:28:39.805812185 +0000 UTC m=+472148.471038380: ☑️ agreed by sdojjy.

@nongfushanquan
Copy link
Contributor

/retest

@CharlesCheung96
Copy link
Contributor Author

/retest

3 similar comments
@CharlesCheung96
Copy link
Contributor Author

/retest

@CharlesCheung96
Copy link
Contributor Author

/retest

@CharlesCheung96
Copy link
Contributor Author

/retest

@ti-chi-bot ti-chi-bot bot merged commit dd3f8df into pingcap:master Nov 26, 2023
14 checks passed
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-6.5: #10161.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 26, 2023
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.1: #10162.

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Nov 26, 2023
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created to branch release-7.5: #10163.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved lgtm needs-cherry-pick-release-6.5 Should cherry pick this PR to release-6.5 branch. needs-cherry-pick-release-7.1 Should cherry pick this PR to release-7.1 branch. needs-cherry-pick-release-7.5 Should cherry pick this PR to release-7.5 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cleanup expired files by day for storage sink
4 participants