cdc: Add integration tests with Debezium

[breezewish]

What problem does this PR solve?

Issue Number: ref #1799

What is changed and how it works?

Currently the integration tests can be run manually. It is not integrated into the CI/CD workflow (will be done in later PRs).

To run the integration test:

1. Compile TiCDC with the Debezium output format:

   cd tiflow
   make cdc

2. Start the Debezium+MySQL+Kafka stack:

   cd tiflow/tests/integration_tests/debezium
   docker compose up

3. Add the Debezium connector, Debezium outputs the changes to output_debezium topic:

   curl -i -X POST \
     -H "Accept:application/json" \
     -H "Content-Type:application/json" \
     localhost:8083/connectors/ --data-binary @- << EOF
   {
     "name": "my-connector",
     "config": {
       "connector.class": "io.debezium.connector.mysql.MySqlConnector",
       "tasks.max": "1",
       "database.hostname": "mysql",
       "database.port": "3306",
       "database.user": "debezium",
       "database.password": "dbz",
       "database.server.id": "184054",
       "topic.prefix": "dbserver1",
       "schema.history.internal.kafka.bootstrap.servers": "kafka:9092",
       "schema.history.internal.kafka.topic": "schemahistory.test",
       "transforms": "x",
       "transforms.x.type": "org.apache.kafka.connect.transforms.RegexRouter",
       "transforms.x.regex": "(.*)",
       "transforms.x.replacement":"output_debezium",
       "binary.handling.mode": "base64",
       "decimal.handling.mode": "double"
     }
   }
   EOF

4. Start TiDB + TiCDC stack:

   tiup playground v7.1.0 --tiflash 0 --ticdc 1 --ticdc.binpath tiflow/bin/cdc

5. Add the Changefeed, which outputs the changes to output_ticdc topic:

   tiup cdc cli changefeed create \
     --server=http://127.0.0.1:8300 \
     --sink-uri="kafka://127.0.0.1:9094/output_ticdc?protocol=debezium&kafka-version=2.4.0"

6. Run the integration test

   go run ./src

The integration test checks whether output number of rows are matching. Example:

And whether output JSONs are matching. Example:

Currently, schema, source and ts_ms is ignored during compare.

Check List

Tests

• Integration test

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

None

[purelind]

/test cdc-integration-kafka-test

[purelind]

/test cdc-integration-kafka-test

[hongyunyan]

/retest

[codecov]

Codecov Report

Merging #10310 (5e08bbf) into master (045c122) will increase coverage by 0.0285%.
Report is 2 commits behind head on master.
The diff coverage is n/a.

Additional details and impacted files

Components	Coverage Δ
cdc	62.0468% <ø> (+0.0419%)	⬆️
dm	51.2902% <ø> (+0.0161%)	⬆️
engine	63.4423% <ø> (+0.0141%)	⬆️

Flag	Coverage Δ
unit	57.6368% <ø> (+0.0285%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #10310        +/-   ##
================================================
+ Coverage   57.6083%   57.6368%   +0.0285%     
================================================
  Files           849        849                
  Lines        126098     126105         +7     
================================================
+ Hits          72643      72683        +40     
+ Misses        48027      47997        -30     
+ Partials       5428       5425         -3     

[hongyunyan]

/test cdc-integration-mysql-test

[hongyunyan]

/test cdc-integration-mysql-test

[hongyunyan]

/test cdc-integration-mysql-test

[hongyunyan]

/test cdc-integration-mysql-test

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, asddongmen

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [3AceShowHand,asddongmen]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2024-02-18 03:28:19.511078056 +0000 UTC m=+154988.258701164: ☑️ agreed by 3AceShowHand.
• 2024-02-19 05:28:36.279245366 +0000 UTC m=+248605.026868461: ☑️ agreed by asddongmen.

[hongyunyan]

/retest