LakeSoul Release v2.4.0 and Python 1.0 Beta

What's New In This Release

1. RBAC support for all query engines. doc
2. Auto cleaning of old compaction data and partition TTL. doc
3. Upgrade Flink version to 1.17 and support row level update/delete in batch sql.
4. Optimize whole database Flink cdc sync throughput by 80%: #307
5. Presto Reader; doc
6. Python reader and integration with PyTorch and HuggingFace. doc

本次更新内容

1. 支持 RBAC 角色权限控制，对所有引擎、所有语言API均有效；文档
2. 自动清理旧的 compaction 数据，支持分区级生命周期（TTL）；文档
3. 升级 Flink 版本到 1.17，并支持批模式下行级别更新和删除；
4. 优化整库同步 Flink 作业，吞吐提升 80%： #307 ；
5. 支持 Presto 读取；文档
6. 支持原生 Python 读取，提供 PyTorch、HuggingFace 的集成。文档

What's Changed

• [NativeIO] Upgrade datafusion to 27 by @xuchen-plus in #282
• [Flink] implement filter pushdown and fix partition pushdown in flink by @xuchen-plus in #287
• Upgrade Flink to 1.17 by @xuchen-plus in #288
• [Python][NativeIO] Add C interface definition by @xuchen-plus in #291
• [NativeIO] update arrow version by @xuchen-plus in #290
• Add Built-in RBAC support by @clouddea in #292
• fix apache license by @clouddea in #293
• [Native-Metadata] Rust implementation of DAO layer by @Ceng23333 in #294
• [Flink] fix jackson-core package in flink by @xuchen-plus in #297
• [Docs] update docs by @xuchen-plus in #298
• [Flink] upgrade flink cdc connector to 2.4 by @xuchen-plus in #303
• clean old compaction data and redundant data by @ChenYunHey in #304
• [Python][Native-Metadata] Python interface of lakesoul metadata by @Ceng23333 in #305
• [Python] C callback with data by @xuchen-plus in #306
• [Python][Dataset] PyArrow and PyTorch dataset api for LakeSoul by @codingfun2022 in #308
• [Flink] rollback flink cdc to 2.3.0 and supplement tables check in benchmark by @F-PHantam in #309
• [Flink] Optimize CDC sink serde with Fury by @xuchen-plus in #307
• [NativeIO] add hdfs feature in lakesoul-io-c by @xuchen-plus in #311
• [Python] exclude partition column at get_arrow_schema_by_table_name by @Ceng23333 in #312
• [Native-Metadata] Retry when native metadata client fail by @Ceng23333 in #313
• [Flink] cdc supplement data delay check mechanism and fix logicallyDropColumn bug by @F-PHantam in #315
• Presto Connector Support by @clouddea in #314
• add scala in common to address build in idea intellij by @xuchen-plus in #316
• [Flink] Ignore exception when hadoop env missing by @xuchen-plus in #317
• [NativeIO] Merge native modules by @Ceng23333 in #318
• bump version to 2.4.0 by @xuchen-plus in #319
• [RBAC] Set hdfs dir owner by @xuchen-plus in #321
• [BugFix]support query metadata with null string by @Ceng23333 in #324
• [Spark] list namespace should return empty array by @xuchen-plus in #323
• [Python][Dataset] Update Python dataset api for LakeSoul by @codingfun2022 in #325
• [Python] Examples using Python API for AI model training by @Ceng23333 in #327
• update docs and readme for release 2.4 by @xuchen-plus in #328
• [Docs] Usage on auto table clean by @ChenYunHey in #326
• [Docs] Add presto connector deployment docs by @xuchen-plus in #329
• [Docs] Add docs for Python and PyTorch by @Ceng23333 in #330
• [Docs] add workspace and rbac docs by @xuchen-plus in #331
• [Bug] turn off native meta query and temporarily disable io prefetch by @F-PHantam in #333
• [Bug]filter should not pushdown before merge on read by @Ceng23333 in #310
• Support view、batch update、batch delete in flink by @moresun in #332
• [Docs ] Refine flink sql and python docs by @xuchen-plus in #337

Full Changelog: https://github.com/lakesoul-io/LakeSoul/commits/v2.4.0