huge logs for "single region event feed disconnected" during initial scan

[fubinzh]

What did you do?

1. Deploy TiDB cluster with 6 TiKV and 2 CDC
2. Stop GC
3. Create blackhole changefeed and pause it
4. Run workload for several days to generate hube incremetal data for cdc t sync
5. Resume cdc changefeed to do initial scan

What did you expect to see?

1. CDC initial scan should work as expected

What did you see instead?

• cdc goroutine number abnormal
  

• Pd grpc scan regions abnormal
  

• huge cdc logs, ~1TB per day
  

Versions of the cluster

Upstream TiKV version (execute tikv-server --version):

/ # /tikv-server -V
TiKV
Release Version:   6.5.3
Edition:           Community
Git Commit Hash:   0578b41fca52aeee5ea5708efd41275165242988
Git Commit Branch: heads/refs/tags/v6.5.3-pr16054-0578b4
UTC Build Time:    2023-11-23 08:19:34
Rust Version:      rustc 1.67.0-nightly (96ddd32c4 2022-11-14)
Enable Features:   pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
Profile:           dist_release

TiCDC version (execute cdc version):

bash-5.1# /cdc version
Release Version: v6.5.3-20231116-255d810
Git Commit Hash: 255d8104c8b9b4f8af191ee69d301ced723ccc3d
Git Branch: heads/refs/tags/v6.5.3-20231116-255d810
UTC Build Time: 2023-11-16 06:35:28
Go Version: go version go1.19.12 linux/amd64
Failpoint Build: false

[fubinzh]

/severity major

[fubinzh]

This issue is not resolved yet.
/reopen

[fubinzh]

/label affects-6.5
/label affects-7.1
/label affects-7.5

[hicqu]

Closed by #10268.