Dagger

Dagger or Data Aggregator is an easy-to-use, configuration over code, cloud-native framework built on top of Apache Flink for stateful processing of real-time streaming data. With Dagger, you don't need to write custom applications or manage resources to process data in real-time. Instead, you can write SQLs to do the processing and analysis on streaming data.

Key Features

Discover why to use Dagger

• Processing: Dagger can transform, aggregate, join and enrich Protobuf data in real-time.
• Scale: Dagger scales in an instant, both vertically and horizontally for high performance streaming sink and zero data drops.
• Extensibility: Add your own sink to dagger with a clearly defined interface or choose from already provided ones.
• Pluggability: Add custom business logic in form of plugins (UDFs, Transformers, Preprocessors and Post Processors) independent of the core logic.
• Metrics: Always know what’s going on with your deployment with built-in monitoring of throughput, response times, errors and more.

What problems Dagger solves?

• Map reduce -> SQL
• Enrichment -> Post Processors
• Aggregation -> SQL, UDFs
• Masking -> Hash Transformer
• Deduplication -> Deduplication Transformer
• Realtime long window processing -> Longbow

To know more, follow the detailed documentation.

Usage

Explore the following resources to get started with Dagger:

• Guides provides guidance on creating Dagger with different sinks.
• Concepts describes all important Dagger concepts.
• Advance contains details regarding advance features of Dagger.
• Reference contains details about configurations, metrics and other aspects of Dagger.
• Contribute contains resources for anyone who wants to contribute to Dagger.
• Usecase describes examples use cases which can be solved via Dagger.

Running locally

Make sure you have Java8 and local kafka-2.4+ setup pre-installed on your local machine.

# Clone the repo
$ git clone https://github.com/odpf/dagger.git  

# Build the jar
$ ./gradlew clean build 

# Configure env variables
$ cat dagger-core/env/local.properties

# Run a Dagger
$ ./gradlew dagger-core:runFlink

Note: Sample configuration for running a basic dagger can be found here. For detailed configurations, refer here.

Find more detailed steps on local setup here.

Running on cluster

Refer here for details regarding Dagger deployment.

Running tests

# Running unit tests
$ ./gradlew clean test

# Run code quality checks
$ ./gradlew checkstyleMain checkstyleTest

# Cleaning the build
$ ./gradlew clean

Contribute

Development of Dagger happens in the open on GitHub, and we are grateful to the community for contributing bug fixes and improvements. Read below to learn how you can take part in improving Dagger.

Read our contributing guide to learn about our development process, how to propose bug fixes and improvements, and how to build and test your changes to Dagger.

To help you get your feet wet and get you familiar with our contribution process, we have a list of good first issues that contain bugs which have a relatively limited scope. This is a great place to get started.

Credits

This project exists thanks to all the contributors.

License

Dagger is Apache 2.0 licensed.