ebs br: resume gc and scheduler when volume snapshots created #5288

WangLe1321 · 2023-09-12T08:32:10Z

What problem does this PR solve?

resume gc and pd scheduler when all the volume snapshots are created
Closes ebs br: shrink the window of pause gc and schedule during backup #5256
send SIGTERM to the BR process when tidb-backup-manager receives SIGTERM
read stderr async in case that the pipe of stderr is full and blocks the BR process
Closes backup job is stuck(snapshot is completed) #5281
The EBS backup task may cause the import task to fail
Closes The EBS backup task may cause the import task to fail #5282

What is changed and how does it work?

data plane:

check if the backupmeta file is existed when the backup enter execute phase. if it is existed, modify backup status VolumeBackupSnapshotsCreated

control plane:
2. if all the backups in data plane has status VolumeBackupSnapshotsCreated, modify resumeGcSchedule field true in the backup that is responsible to pause gc and pd schduler.

data plane:
3. if resumeGcSchedule field is true, delete the initialize pod and modify backup status VolumeBackupInitializeComplete

control plane:
5. if the backup with resumeGcSchedule: true has status VolumeBackupInitializeComplete, modify volume backup status SnapshotsCreated. It means all the volume snapshots are created and gc and pd schedulers are resumed.

Code changes

Has Go code change
Has CI related scripts change

Tests

Unit test
E2E test
Manual test
No code

Side effects

Breaking backward compatibility
Other side effects:

Related changes

Need to cherry-pick to the release branch
Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

codecov-commenter · 2023-09-14T09:36:52Z

Codecov Report

Merging #5288 (3ec02e0) into master (2c3cd13) will increase coverage by 2.29%.
The diff coverage is 29.74%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5288      +/-   ##
==========================================
+ Coverage   57.60%   59.90%   +2.29%     
==========================================
  Files         252      255       +3     
  Lines       30461    34113    +3652     
==========================================
+ Hits        17548    20436    +2888     
- Misses      11182    11832     +650     
- Partials     1731     1845     +114

Flag	Coverage Δ
e2e	`29.44% <4.76%> (?)`
unittest	`57.48% <31.42%> (-0.13%)`	⬇️

WangLe1321 · 2023-09-15T03:16:19Z

/test pull-e2e-kind pull-e2e-kind-across-kubernetes pull-e2e-kind-basic

ti-chi-bot · 2023-09-15T03:16:20Z

@WangLe1321: The label(s) /pull-e2e-kind, /pull-e2e-kind-across-kubernetes, /pull-e2e-kind-basic cannot be applied, because the repository doesn't have them.

In response to this:

/test pull-e2e-kind pull-e2e-kind-across-kubernetes pull-e2e-kind-basic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-09-15T03:16:21Z

@WangLe1321: No presubmit jobs available for pingcap/tidb-operator@master

In response to this:

/test pull-e2e-kind pull-e2e-kind-across-kubernetes pull-e2e-kind-basic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

WangLe1321 · 2023-09-15T03:40:04Z

/test pull-e2e-kind-across-kubernetes pull-e2e-kind-basic

ti-chi-bot · 2023-09-15T03:40:06Z

@WangLe1321: The label(s) /pull-e2e-kind-across-kubernetes, /pull-e2e-kind-basic cannot be applied, because the repository doesn't have them.

In response to this:

/test pull-e2e-kind-across-kubernetes pull-e2e-kind-basic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-09-15T03:40:06Z

@WangLe1321: No presubmit jobs available for pingcap/tidb-operator@master

In response to this:

/test pull-e2e-kind-across-kubernetes pull-e2e-kind-basic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

WangLe1321 · 2023-09-15T03:49:45Z

/test pull-e2e-kind-serial pull-e2e-kind-br

ti-chi-bot · 2023-09-15T03:49:46Z

@WangLe1321: The label(s) /pull-e2e-kind-serial, /pull-e2e-kind-br cannot be applied, because the repository doesn't have them.

In response to this:

/test pull-e2e-kind-serial pull-e2e-kind-br

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-09-15T03:49:47Z

@WangLe1321: No presubmit jobs available for pingcap/tidb-operator@master

In response to this:

/test pull-e2e-kind-serial pull-e2e-kind-br

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BornChanger · 2023-09-18T11:11:56Z

/test pull-e2e-kind-serial

ti-chi-bot · 2023-09-18T11:11:58Z

@BornChanger: The label(s) /pull-e2e-kind-serial cannot be applied, because the repository doesn't have them.

In response to this:

/test pull-e2e-kind-serial

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-09-18T11:11:58Z

@BornChanger: No presubmit jobs available for pingcap/tidb-operator@master

In response to this:

/test pull-e2e-kind-serial

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BornChanger · 2023-09-21T10:13:51Z

/label ok-to-test

ti-chi-bot · 2023-09-21T10:13:52Z

@BornChanger: The label(s) /ok-to-test cannot be applied, because the repository doesn't have them.

In response to this:

/label ok-to-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

BornChanger · 2023-09-21T10:14:15Z

/test pull-e2e-kind-across-kubernetes

ti-chi-bot · 2023-09-21T10:14:17Z

@BornChanger: The label(s) /pull-e2e-kind-across-kubernetes cannot be applied, because the repository doesn't have them.

In response to this:

/test pull-e2e-kind-across-kubernetes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-09-21T10:14:31Z

@BornChanger: No presubmit jobs available for pingcap/tidb-operator@master

In response to this:

/test pull-e2e-kind-across-kubernetes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

purelind · 2023-09-21T10:18:35Z

/test pull-e2e-kind-basic

ti-chi-bot · 2023-09-21T10:18:37Z

@purelind: The label(s) /pull-e2e-kind-basic cannot be applied, because the repository doesn't have them.

In response to this:

/test pull-e2e-kind-basic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

ti-chi-bot · 2023-09-21T10:18:37Z

@purelind: No presubmit jobs available for pingcap/tidb-operator@master

In response to this:

/test pull-e2e-kind-basic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BornChanger · 2023-09-21T10:24:50Z

/run-e2e-tests

purelind · 2023-09-21T10:26:15Z

/run-e2e-tests

BornChanger · 2023-09-21T13:50:44Z

/test-pull-e2e-kind-tikv-scale-simultaneously

WangLe1321 · 2023-09-22T02:19:53Z

/run-pull-e2e-kind

WangLe1321 · 2023-09-22T02:20:06Z

/run-pull-e2e-kind-br

WangLe1321 · 2023-09-22T02:20:32Z

/run-pull-e2e-kind-tikv-scale-simultaneously

WangLe1321 · 2023-09-22T02:24:02Z

/run-pull-e2e-kind-tikv-scale-simultaneously

ti-chi-bot · 2023-09-22T06:34:46Z

[LGTM Timeline notifier]

Timeline:

2023-09-20 02:45:48.27226362 +0000 UTC m=+658314.239851671: ☑️ agreed by BornChanger.
2023-09-22 06:34:45.328215977 +0000 UTC m=+844851.295804028: ✖️🔁 reset by WangLe1321.

ti-chi-bot · 2023-09-22T06:34:47Z

New changes are detected. LGTM label has been removed.

WangLe1321 · 2023-09-22T10:45:41Z

/rtest

WangLe1321 · 2023-09-22T10:45:47Z

/retest

BornChanger · 2023-09-23T12:29:08Z

/test pull-e2e-kind

BornChanger · 2023-09-23T12:29:15Z

/test pull-e2e-kind-br

ti-chi-bot · 2023-09-23T12:29:24Z

@BornChanger: No presubmit jobs available for pingcap/tidb-operator@master

In response to this:

/test pull-e2e-kind

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ti-chi-bot · 2023-09-23T12:29:31Z

@BornChanger: No presubmit jobs available for pingcap/tidb-operator@master

In response to this:

/test pull-e2e-kind-br

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

BornChanger · 2023-09-25T06:57:35Z

/cherry-pick release-1.5

ti-chi-bot · 2023-09-25T06:58:12Z

@BornChanger: new pull request created to branch release-1.5: #5298.

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

…#5298) Co-authored-by: WangLe1321 <[email protected]> Co-authored-by: csuzhangxc <[email protected]>

br: modify to resume gc and scheduler when volume snapshots created

1898c8e

ti-chi-bot bot requested review from howardlau1999 and lichunzhu September 12, 2023 08:32

ti-chi-bot bot added the size/XL label Sep 12, 2023

br: modify unit test of federation volume backup

6d1faff

BornChanger added the area/ebs-br label Sep 13, 2023

BornChanger mentioned this pull request Sep 13, 2023

EBS BR support to multiple k8s cluster TiDB #5003

Open

BornChanger changed the title ~~br: modify to resume gc and scheduler when volume snapshots created~~ br: resume gc and scheduler when volume snapshots created Sep 13, 2023

BornChanger changed the title ~~br: resume gc and scheduler when volume snapshots created~~ ebs br: resume gc and scheduler when volume snapshots created Sep 13, 2023

BornChanger requested review from BornChanger and csuzhangxc and removed request for howardlau1999 and lichunzhu September 13, 2023 23:23

WangLe1321 added 2 commits September 14, 2023 16:59

br: propagate SIGTERM to tidb-backup-manager process

6dc4ccb

br: modify execute tidb-backup-manager async

3190af7

WangLe1321 added 2 commits September 14, 2023 18:12

br: fix not waiting subprocess bug

ace08da

br: remove context from exec

5ee7a57

br: modify only backup init pod using gracefully shutting down

3ec02e0

ti-chi-bot bot removed the lgtm label Sep 22, 2023

csuzhangxc merged commit 4df3649 into pingcap:master Sep 25, 2023
4 checks passed

ti-chi-bot mentioned this pull request Sep 25, 2023

ebs br: resume gc and scheduler when volume snapshots created (#5288) #5298

Merged

10 tasks

csuzhangxc added a commit that referenced this pull request Sep 25, 2023

ebs br: resume gc and scheduler when volume snapshots created (#5288) (…

3d41f93

…#5298) Co-authored-by: WangLe1321 <[email protected]> Co-authored-by: csuzhangxc <[email protected]>

ebs br: resume gc and scheduler when volume snapshots created #5288

ebs br: resume gc and scheduler when volume snapshots created #5288

Conversation

WangLe1321 commented Sep 12, 2023 • edited by BornChanger Loading

What problem does this PR solve?

What is changed and how does it work?

Code changes

Tests

Side effects

Related changes

Release Notes

codecov-commenter commented Sep 14, 2023 • edited Loading

Codecov Report

WangLe1321 commented Sep 15, 2023

ti-chi-bot bot commented Sep 15, 2023

ti-chi-bot bot commented Sep 15, 2023

WangLe1321 commented Sep 15, 2023

ti-chi-bot bot commented Sep 15, 2023

ti-chi-bot bot commented Sep 15, 2023

WangLe1321 commented Sep 15, 2023

ti-chi-bot bot commented Sep 15, 2023

ti-chi-bot bot commented Sep 15, 2023

BornChanger commented Sep 18, 2023

ti-chi-bot bot commented Sep 18, 2023

ti-chi-bot bot commented Sep 18, 2023

BornChanger commented Sep 21, 2023

ti-chi-bot bot commented Sep 21, 2023

BornChanger commented Sep 21, 2023

ti-chi-bot bot commented Sep 21, 2023

ti-chi-bot bot commented Sep 21, 2023

purelind commented Sep 21, 2023

ti-chi-bot bot commented Sep 21, 2023

ti-chi-bot bot commented Sep 21, 2023

BornChanger commented Sep 21, 2023

purelind commented Sep 21, 2023

BornChanger commented Sep 21, 2023

WangLe1321 commented Sep 22, 2023

WangLe1321 commented Sep 22, 2023

WangLe1321 commented Sep 22, 2023

WangLe1321 commented Sep 22, 2023

ti-chi-bot bot commented Sep 22, 2023

[LGTM Timeline notifier]

ti-chi-bot bot commented Sep 22, 2023

WangLe1321 commented Sep 22, 2023

WangLe1321 commented Sep 22, 2023

BornChanger commented Sep 23, 2023

BornChanger commented Sep 23, 2023

ti-chi-bot bot commented Sep 23, 2023

ti-chi-bot bot commented Sep 23, 2023

BornChanger commented Sep 25, 2023

ti-chi-bot commented Sep 25, 2023

WangLe1321 commented Sep 12, 2023 •

edited by BornChanger

Loading

codecov-commenter commented Sep 14, 2023 •

edited

Loading