-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
99 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,99 @@ | ||
# sparkle | ||
# Sparkle ✨ | ||
|
||
**Sparkle** is a meta-framework built on top of [Apache | ||
Spark](https://spark.apache.org/), designed to streamline data | ||
engineering workflows and accelerate the delivery of data | ||
products. Developed by [**DataChef**](https://datachef.co), Sparkle | ||
focuses on three main areas: | ||
|
||
1. **Improving Developer Experience (DevEx) 🚀** | ||
2. **Reducing Time to Market ⏱️** | ||
3. **Easy Maintenance 🔧** | ||
|
||
With these goals in mind, Sparkle has enabled DataChef to deliver | ||
functional data products from day one, allowing for seamless handovers | ||
to internal teams. | ||
|
||
## Key Features | ||
|
||
### 1. Improved Developer Experience 🚀 | ||
|
||
Sparkle enhances the developer experience by abstracting away | ||
non-business-critical aspects of Spark application development. It | ||
achieves this through: | ||
|
||
- **Sophisticated Configuration Mechanism**: Simplifies the setup and | ||
configuration of Spark applications, allowing developers to focus | ||
solely on business logic. | ||
- **Automatic Functional Tests 🧪**: Generates tests for each | ||
application automatically, based on predefined input and output | ||
fixtures. This ensures that the application behaves as expected | ||
without requiring extensive manual testing. | ||
|
||
### 2. Reduced Time to Market ⏱️ | ||
|
||
Sparkle significantly reduces the time to market by automating the | ||
deployment and testing processes. This allows data engineers to | ||
concentrate exclusively on developing the business logic, with all | ||
other aspects handled by Sparkle: | ||
|
||
- **Automated Testing ✅**: Ensures that all applications are robust | ||
and ready for deployment without manual intervention. | ||
- **Seamless Deployment 🚢**: Automates the deployment pipeline, | ||
reducing the time needed to bring new data products to market. | ||
|
||
### 3. Enhanced Maintenance 🔧 | ||
|
||
Sparkle simplifies maintenance through heavy testing and abstraction | ||
of non-business functional requirements. This provides a reliable and | ||
trustworthy system that is easy to maintain: | ||
|
||
- **Abstraction of Non-Business Logic 📦**: By focusing on business | ||
logic, Sparkle minimizes the complexity associated with maintaining | ||
Spark applications. | ||
- **Heavily Tested Framework 🔍**: All non-business functionalities | ||
are thoroughly tested, reducing the risk of bugs and ensuring a | ||
stable environment for data applications. | ||
|
||
## How It Works 🛠️ | ||
|
||
The Sparkle framework operates on a principle similar to Function as a | ||
Service (FaaS). Developers can instantiate a Sparkle application that | ||
takes a list of input DataFrames and focuses solely on transforming | ||
these DataFrames according to the business logic. The Sparkle | ||
application then automatically writes the output of this | ||
transformation to the desired destination. | ||
|
||
## Getting Started 🚀 | ||
|
||
Sparkle is currently under heavy development, and we are continuously | ||
working on improving and expanding its capabilities. | ||
|
||
To stay updated on our progress and access the latest information, | ||
follow us on [LinkedIn](https://nl.linkedin.com/company/datachefco) | ||
and [GitHub](https://github.com/DataChefHQ/Sparkle). | ||
|
||
## Contributing 🤝 | ||
|
||
We welcome contributions from the community! If you're interested in | ||
contributing to Sparkle, please check our [GitHub | ||
repository](https://github.com/DataChefHQ/Sparkle) for more details on | ||
how you can get involved. | ||
|
||
## License 📄 | ||
|
||
Sparkle is licensed under the Apache v2.0 License. See the | ||
[LICENSE](LICENSE) file for more details. | ||
|
||
## Contact 📬 | ||
|
||
For more information, questions, or feedback, feel free to reach out | ||
to us on [LinkedIn](https://nl.linkedin.com/company/datachefco) or | ||
open an issue on our | ||
[GitHub](https://github.com/DataChefHQ/sparkle/issues) repository. | ||
|
||
--- | ||
|
||
Thank you for your interest in Sparkle! We're excited to have you join | ||
us on this journey to revolutionize data engineering with Apache | ||
Spark. 🎉 |