LakeSoul Release v2.3.0

[xuchen-plus]

v2.3.0 Release Notes

This is the first release after LakeSoul donated to Linux Foundation AI & Data. This release contains the following major new features:

1. Flink Connector for Flink SQL/Table API to read or write LakeSoul in both batch and streaming mode, with the supports of Flink Changelog Stream semantics and row-level upsert and delete. See docs Flink Connector.
2. Flink CDC Ingestion refactored to infer new tables and schema changes automatically from messages. This enables simpler CDC stream ingestion job development for any kinds of database or message queues.
3. Global automatic compaction service. See docs Auto Compaction Service.

更新日志

这是 LakeSoul 捐赠给 Linux Foundation AI & Data 后的第一个发布版本。该版本包含以下重要更新：

1. 全面支持 Flink SQL/Table API. LakeSoul 支持 Flink 流、批读写。流式读写完整支持 Flink Changelog 语义，支持行级别流式增删改。参考文档
2. Flink CDC 整库同步重构，支持从消息中自动推断新表和 schema 变更。能够更简单的开发 CDC 入湖作业并支持消费任意数据库 CDC 流或消息队列流。
3. 全局自动 Compaction 服务。参考文档：LakeSoul 全局自动压缩服务使用方法

What's Changed

• [NativeIO] Native io misc improvements by @dmetasoul01 in [NativeIO] Native io misc improvements #190
• optimize filesForScan by @F-PHantam in optimize filesForScan #192
• Add Definition Comments for com.dmetasoul.lakesoul.meta.entity by @YuChangHui in Add Definition Comments for com.dmetasoul.lakesoul.meta.entity #193
• Implement Delta Join Interfaces for LakeSoulTable by @YuChangHui in Implement Delta Join Interfaces for LakeSoulTable #184
• [Flink] pack paranamer to flink release jar by @dmetasoul01 in [Flink] pack paranamer to flink release jar #196
• [NativeIO] use tcmalloc as global allocator by @xuchen-plus in [NativeIO] use tcmalloc as global allocator #204
• [NativeIO] fix memory leak in native reader by @xuchen-plus in [NativeIO] fix memory leak in native reader #209
• [Flink] avoid cast global parameter to ParameterTool by @xuchen-plus in [Flink] avoid cast global parameter to ParameterTool #207
• migrate arrow-rs and datafusion deps to new org by @xuchen-plus in migrate arrow-rs and datafusion deps to new org #211
• Implement Global Automatic Disaggregated Compaction Service by @F-PHantam in Implement Global Automatic Disaggregated Compaction Service #212
• Implement Flink ScanTableSource and LookupTableSource by @YuChangHui in Implement Flink ScanTableSource and LookupTableSource #213
• fix data type timestamp with zone by @lypnaruto in fix data type timestamp with zone #215
• [NativeIO]throw execption when LakeSoulArrowReader.hasNext by @Ceng23333 in [NativeIO]throw execption when LakeSoulArrowReader.hasNext #217
• [NativeIO]add rust clippy workflow && fix clippy error/warn by @Ceng23333 in [NativeIO]add rust clippy workflow && fix clippy error/warn #219
• add flink sql submitter([Flink] SQL Application Submitter #199) by @Hades-888 in add flink sql submitter(#199) #221
• Update readme by @xuchen-plus in Update readme #222
• bump version to 2.3.0 by @xuchen-plus in bump version to 2.3.0 #223
• update github links by @xuchen-plus in update github links #224
• fix bug: requested file schema no change in stream task by @F-PHantam in fix bug: requested file schema no change in stream task #226
• [Flink]LakeSoulCatalog::listTables: list tableName instead of tablePath by @Ceng23333 in [Flink]LakeSoulCatalog::listTables: list tableName instead of tablePath #227
• [Flink]fix parse error of LogicalTypeRoot::Date by @Ceng23333 in [Flink]fix parse error of LogicalTypeRoot::Date #228
• [NativeIO]panic when target datatype and source datatype mismatch by @Ceng23333 in [NativeIO]panic when target datatype and source datatype mismatch #214
• [Flink]support flink decimal by @Ceng23333 in [Flink]support flink decimal #232
• update LakeSoulTableSource.getChangelogMode by @Ceng23333 in update LakeSoulTableSource.getChangelogMode #231
• [NativeIO]fix clippy warning by @Ceng23333 in [NativeIO]fix clippy warning #230
• Fix hash bucket num by @xuchen-plus in Fix hash bucket num #233
• [Flink]add batch in flink sql submitter by @Hades-888 in [Flink]add batch in flink sql submitter #234
• disable tcmalloc by @xuchen-plus in disable tcmalloc #235
• [Project] add lakesoul project website code by @xuchen-plus in [Project] add lakesoul project website code #237
• update load flink sql from hdfs in yarn application by @Hades-888 in update load flink sql from hdfs in yarn application #238
• [Flink]add Maven-test CI for lakesoul-flink by @lypnaruto in [Flink]add Maven-test CI for lakesoul-flink #239
• Add cross build for native io by @xuchen-plus in Add cross build for native io #241
• [Project] disable git lfs by @xuchen-plus in [Project] disable git lfs #243
• fix bugs for same bucket readed by differnet stream tasks by @moresun in fix bugs for same bucket readed by differnet stream tasks #245
• [Project] Add pr checks and deployment actions by @xuchen-plus in [Project] Add pr checks and deployment actions #244
• [Flink]fix FlinkDatatype::timestamp_ltz zone conversion && support FlinkDatatype::timestamp by @Ceng23333 in [Flink]fix FlinkDatatype::timestamp_ltz zone conversion && support FlinkDatatype::timestamp #246
• Prepare meta in maven test by @xuchen-plus in Prepare meta in maven test #247
• [Flink]Fix LookupSource FS configuration setting by @Ceng23333 in [Flink]Fix LookupSource FS configuration setting #248
• LakeSoul mysql cdc convert Datatype::datetime to timestamp with timezone by @F-PHantam in LakeSoul mysql cdc convert Datatype::datetime to timestamp with timezone #249
• [Spark] Fix compatibility with spark 3.3.2 by @xuchen-plus in [Spark] Fix compatibility with spark 3.3.2 #251
• add flink source and sink ci test by @F-PHantam in add flink source and sink ci test #252
• [Flink] fix wrong logging config file in flink test by @xuchen-plus in [Flink] fix wrong logging config file in flink test #253
• [Flink] Move partition column fill to native io by @xuchen-plus in [Flink] Move partition column fill to native io #254
• Fix datatype conversion from flink to spark by @Ceng23333 in Fix datatype conversion from flink to spark #255
• [Flink] Add source failover test cases by @xuchen-plus in [Flink] Add source failover test cases #256
• [Flink] LakeSoulSinkGlobalCommitter by @Ceng23333 in [Flink] LakeSoulSinkGlobalCommitter #257
• add LAKESOUL_PARTITION_SPLITTER as constant by @Ceng23333 in add LAKESOUL_PARTITION_SPLITTER as constant #260
• remove guava and commons-lang in common module by @xuchen-plus in remove guava and commons-lang in common module #261
• Modify mysqlcdc sort key generation way by @F-PHantam in Modify mysqlcdc sort key generation way #263
• [Flink] Add sink failover test cases by @Ceng23333 in [Flink] Add sink failover test cases #259
• [Flink] Fix flink reader npe by @xuchen-plus in [Flink] Fix flink reader npe #265
• [Flink]complete test options of sink fail tests by @Ceng23333 in [Flink]complete test options of sink fail tests #266
• Refine meta partition values by @xuchen-plus in Refine meta partition values #267
• [Flink]Check schema migration at GlobalCommitter by @Ceng23333 in [Flink]Check schema migration at GlobalCommitter #269
• Fix meta exception handling by @xuchen-plus in Fix meta exception handling #270
• Update website and readme for 2.3.0 release by @xuchen-plus in Update website and readme for 2.3.0 release #271

---

This discussion was created from the release LakeSoul Release v2.3.0.