Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Sort] Add batch end-to-end test for Kafka v1.18 connector #11573

Open
2 tasks done
PeterZh6 opened this issue Dec 2, 2024 · 1 comment
Open
2 tasks done

Comments

@PeterZh6
Copy link
Contributor

PeterZh6 commented Dec 2, 2024

Description

Parent Issue #11419
Related Issue #11494

Flink uses stream mode as default mode of execution. Current end-to-end test introduced by #11494 adopts stream mode. Additional batch-executed end-to-end test need to be enforced to better verify the requirements for #11419, and potentially for all the sub-issues under #11417.

Use case

No response

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

@PeterZh6
Copy link
Contributor Author

PeterZh6 commented Dec 4, 2024

While working on this, I started to wonder about the best approach for conducting such a test:

Should I simply use Flink's batch execution mode with the command-line arguments (i.e., --execution.mode batch) when submitting the Flink job, and test it that way?

Or do I need to tailor the test for each connector (in this case, Kafka) since different connectors can behave differently in batch mode?

I’m uncertain whether a one-size-fits-all approach will work, or if I need to adjust configurations or setup steps specific to Kafka or the InLong Sort connector.

Has anyone faced this scenario or can provide insights on how best to test batch support with Flink connectors?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant