Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Hadoop DSL Canonical Compiler and Azkaban Reverse Job Compiler #101

Open
convexquad opened this issue Sep 12, 2016 · 1 comment

Comments

@convexquad
Copy link
Contributor

The idea is to implement a Hadoop DSL compiler that actually writes out formatted Hadoop DSL. This would be necessary for a Azkaban job files -> Hadoop DSL reverse compiler.

In people's Hadoop DSL files they will have all kinds of Groovy that is actually evaluated at Gradle build time. They end up with a Hadoop DSL structure in memory. The canonical compiler would spit out the Hadoop DSL structure as Hadoop DSL itself, but obviously with no Groovy logic in it.

Once the canonical compiler is in place, the task of implementing an Azkaban reverse compiler would simply consist of a processor for the Azkaban .job and .properties files that builds up the corresponding Hadoop DSL structure in memory and then just writes it out using the canonical compiler.

@convexquad
Copy link
Contributor Author

The reverse compiler would be a more challenging longer-term task. It should definitely be broken up into these two parts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant