v2.1.0

[dmetasoul01]

v2.1.0 Release Notes

LakeSoul 2.1.0 brings new Flink CDC sink implementation which supports all tables (with different schemas) in one entire MySQL database sync in one Flink job, automatic schema sync and evolution, automatic new table creation and exactly once guarantee. The currently supported flink version is 1.14.

In 2.1.0 we also reimplement Spark catalog so that it could be used as a standalone catalog rather than a session catalog extension. This change is to avoid some inconsistencies in Spark's v2 table commands, e.g. show tables cannot support v2 tables until 3.3.

Packages for Spark and Flink are separated into two maven submodules. The maven coordinates are com.dmetasoul:lakesoul-spark:2.1.0-spark-3.1.2 and com.dmeatsoul:lakesoul-flink:2.1.0-flink-1.14. All the required transitive dependencies have already been shaded into the released jars.

v2.1.0 发布日志

LakeSoul 2.1.0 增加了全新的 Flink CDC Sink 功能，支持 MySQL 数据库整库千表（支持不同 schema）同步，自动 Schema 变更同步，自动新表感知和严格一次（Exactly Once）语义保证。

Spark 支持部分重写了 Catalog 的实现，使得 Catalog 可以作为非 Session Catalog 扩展使用，主要目的是规避 Spark 在 3.3 版本之前，一些 DDL Command 不支持 V2 表的问题。

Spark 和 Flink 分别拆分成了两个 Maven 子模块。在工程中引用的 Maven 坐标分别是 com.dmetasoul:lakesoul-spark:2.1.0-spark-3.1.2 and com.dmeatsoul:lakesoul-flink:2.1.0-flink-1.14。他们各自的依赖库已经通过 shade 的方式打包到了发布的 jar 包中。

Merged Pull Requests

• CDC support v1: add table property to identify change kind column by @dmetasoul01 in CDC support v1: add table property to identify change kind column #1
• Cdc support v2 by @moresun in Cdc support v2 #3
• support merge into sql when can be converted to upsert by @dmetasoul01 in support merge into sql when can be converted to upsert #4
• Optimize duplicate tests and code by @dmetasoul01 in Optimize duplicate tests and code #6
• support create hash partitioned table by sql by @dmetasoul01 in support create hash partitioned table by sql #7
• remove cdc filter from mergescan by @moresun in remove cdc filter from mergescan #9
• fix build error and some coding styles by @bakey in fix build error and some coding styles #10
• Update README.md by @moresun in Update README.md #13
• add a cdc sink example by @dmetasoul01 in add a cdc sink example #17
• update all links in readme to relative by @dmetasoul01 in update all links in readme to relative #18
• [Doc] add cdc cn doc by @dmetasoul01 in [Doc] add cdc cn doc #19
• Bump fastjson from 1.2.75 to 1.2.83 by @dependabot in Bump fastjson from 1.2.75 to 1.2.83 #38
• Catalog refactor by @dmetasoul01 in Catalog refactor #45
• Bump mysql-connector-java from 8.0.19 to 8.0.28 by @dependabot in Bump mysql-connector-java from 8.0.19 to 8.0.28 #46
• Bump postgresql from 42.2.14 to 42.3.3 by @dependabot in Bump postgresql from 42.2.14 to 42.3.3 #47
• bump version to 2.0.0 by @dmetasoul01 in bump version to 2.0.0 #48
• fix maven packaging by @dmetasoul01 in fix maven packaging #55
• Feature flink order sink by @YangZH-v2 in Feature flink order sink #56
• add parquet-column dependency fix localEnv unable run bug by @YangZH-v2 in add parquet-column dependency fix localEnv unable run bug #64
• support exactly once semantics for flink write by @YangZH-v2 in support exactly once semantics for flink write #65
• fix filter bug when cdc column is not used by @F-PHantam in fix filter bug when cdc column is not used #68
• Align hash bucket and sort logic in flink with spark Align hash bucket and sort logic in Flink with Spark #60 by @YangZH-v2 in Align hash bucket and sort logic in flink with spark #60 #69
• Split submodules for maven project by @F-PHantam in Split submodules for maven project #70
• Bump postgresql from 42.3.3 to 42.4.1 in /lakesoul-common by @dependabot in Bump postgresql from 42.3.3 to 42.4.1 in /lakesoul-common #71
• add MergeNonNullOp for merge operator by @moresun in add MergeNonNullOp for merge operator #73
• add docker compose for local test. fix maven install gpg signing by @dmetasoul01 in add docker compose for local test. fix maven install gpg signing #76
• clean up unused code by @dmetasoul01 in clean up unused code #77
• fix MultiPartitionMergeBucketScan bug by @F-PHantam in fix MultiPartitionMergeBucketScan bug #81
• Fix flink cdc write event order by @dmetasoul01 in Fix flink cdc write event order #82
• supports database(namespace) & support mysql cdc using flink by @Ceng23333 in supports database(namespace) & support mysql cdc using flink #85
• Bump snakeyaml from 1.30 to 1.31 in /lakesoul-spark by @dependabot in Bump snakeyaml from 1.30 to 1.31 in /lakesoul-spark #88
• Support multiple tables sink for Flink CDC by @dmetasoul01 in Support multiple tables sink for Flink CDC #86
• flink cdc task add argument serverTimeZone by @F-PHantam in flink cdc task add argument serverTimeZone #90
• Fix maven dependency by @dmetasoul01 in Fix maven dependency #91

New Contributors

• @dmetasoul01 made their first contribution in CDC support v1: add table property to identify change kind column #1
• @moresun made their first contribution in Cdc support v2 #3
• @bakey made their first contribution in fix build error and some coding styles #10
• @dependabot made their first contribution in Bump fastjson from 1.2.75 to 1.2.83 #38
• @YangZH-v2 made their first contribution in Feature flink order sink #56
• @F-PHantam made their first contribution in fix filter bug when cdc column is not used #68
• @Ceng23333 made their first contribution in supports database(namespace) & support mysql cdc using flink #85

Full Changelog: https://github.com/meta-soul/LakeSoul/commits/2.1.0

---

This discussion was created from the release v2.1.0.