-
Notifications
You must be signed in to change notification settings - Fork 188
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Coral-Incremental] Incremental plan generation for RelNode incremental rewrite #520
base: master
Are you sure you want to change the base?
Conversation
Subquery
* - For other RelNodes: recursively processing their inputs to ensure uniformity. | ||
* <p> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not clear what this achieves.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the example below show what it's doing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am referring to "to ensure uniformity". The idea about the whole method is to ensure uniformity. Making it exclusive to "other RelNodes" does not make sense.
I think overall, the PR could benefit from more extensive documentation. Current documentation on different methods is a good start. Each documentation should be expanded with concrete running example with actual table names in the plan, input and output. Each method responsibility throughout the transformation process could be illustrated by this running example. If you add the table names to your current example, it will make the documentation much more clear and concrete by giving examples of the input and output of each method. |
What changes are proposed in this pull request, and why are they necessary?
Introduction: The goal of this PR is to introduce Incremental Plan Generation for Coral-Incremental. This feature will enable the generation of multiple incremental plans for a given RelNode (a logical plan) based on the number of sub-queries that will be computed incrementally. The number of generated plans will match the number of sub-queries in a SQL query. Each logical plan in these generated plans will update the materialized view with its results.
For example, in a multi-table join, we can choose between batch and incremental execution stages. Consider the following three-table join:
We could generate plans as follows:
Each generated plan will be represented as a List, where each RelNode indicates whether its corresponding sub-query will be executed incrementally or in batch mode.
Some important changes:
RelNodeGenerationTransformer
to rewrite incremental format RelNodes by materializing the join subqueries, and combine them to generate different complete plans.RelNodeGenerationTransformer
:findJoinNeedsProject
: Converts allRelNodes
into a uniform format for future incremental plan generation.uniformFormat
: Ensures consistent formatting ofRelNodes
.convertRelPrev
method which could convert the tables of aRelNode
into '_prev' format, thus distinguishing from the original one.How was this patch tested?
Unit test