Knowledge Sharing (KS) - Building real-time data pipelines

References

• GitHub - Data Engineering Helpers - KS - Real-time data pipelines (this Git repository)

Quick starter with Docker, Kafka, Delta and Polars

• Medium - Building a real-time data pipeline
  • GitHub companion repository - Building a streaming data pipeline

Delta-Lake

• Delta.io blog - Delta Lake - State of the Project, Part 2, April 2024
  • Delta.io blog - Delta Lake - State of the Project, Part 1, March 2024
• Delta.io docs - Quick start - Python

Delta-RS and Delta Kernel

• Delta.io blog - Delta Kernel, Building Delta Lake connectors, made simple, June 2024
  • Docs.RS - Delta Kernel
• GitHub - Delta-Kernel-RS
• GitHub - Delta-RS

Connectors for DuckDB and Polars

• DuckDB docs - Native Delta Lake Support in DuckDB, June 2024
• Delta.io blog - Reading Delta Lake Tables into Polars DataFrames, December 2022