Skip to content

Latest commit

 

History

History
40 lines (28 loc) · 2.11 KB

README.md

File metadata and controls

40 lines (28 loc) · 2.11 KB
Raw

Data Engineering Practice

A simple project to practice data engineering skills & migrating python projects to rust.

The project is purely created for learning Rust by projects.

Who am I?

My name is Kaan. I am a software developer at GAİN, writing flutter. I like to experiment and learn new things. Apart from that, I am passionate about Data-engineering & open-source. I am also a big fan of Rust. I am trying to learn Rust through projects. I am open to any kind of feedback.

Used Libraries

Rust

  • clap for parsing command line arguments.
  • zld for faster linking to the binary (brew install michaeleisel/zld/zld).
  • bacon background rust code checker.
  • git-cliff for generating changelog from conventional git commits.
  • tokio for the async runtime environment.
  • reqwest high-level HTTP client.
  • color-eyre An error report handler for panics and colorful, consistent, and well-formatted error reporter.
  • sqlx SQLx is an async, pure Rust† SQL crate featuring compile-time checked queries without a DSL. It is not an ORM tool.
  • polars Polars is a blazingly fast DataFrames library implemented in Rust.

Python

  • sqlalchemy SQL toolkit and object-relational mapper.
  • pandas Tool for data manipulation and analysis.

TODO

  • Find a way to manage versioning for cargo file.
  • Add environment management for the project.
  • Add unit tests for both scripts.
  • Add CI/CD pipelines for both scripts.
  • Create badges for the project.
  • Experiment with dockerizing the project.
  • Experiment with docker-compose & terraform.

You could reach me on Twitter to discuss the project & help me to improve myself & the project.