This repository has been archived by the owner on Feb 20, 2023. It is now read-only.
Index Scan + Index Join Limit #1422
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thepinetree
added
in-progress
thepinetree
added
ready-for-review
jkosh44
reviewed
Dec 31, 2020
| - [ ] Add information for subquery transformation above | ||
| - [ ] Add information about aggregations in `SELECT` here | ||
| - [ ] Add information about `SelectDistinct` in `SELECT` here | ||
| - If a `LIMIT` exists in the select statement, the value of the limit and associated offset, along with any `ORDER_BY` clauses, are stored in a `LogicalLimit` node that is added as the parent of the existing `output_expr_` |
There was a problem hiding this comment.
I think it would be helpful here to briefly explain why ORDER_BY information is stored within a LIMIT operator
jkosh44
reviewed
Dec 31, 2020
| - `GroupExpression`: a class to track a specific operator (logical or physical) in a group which tracks the transaction, group number, whether the stats have been derived, the lowest cost satisfying the required properties of an expression, and the details of the expression. | ||
| - `Group`: a class to collect group expressions representing logically equivalent expressions including logical and physical transformations. | ||
| - `LogicalExpression`: a group expression which represents a logical expression i.e. the logical operator that specifies the general behavior of a node | ||
| - `PhysicalExpression`: a group expression which represents a logical expression i.e. the physical operator that specifies the implementation details of the expression |
There was a problem hiding this comment.
Did you mean to say "logical expression" here?
There was a problem hiding this comment.
Which line is this about?
There was a problem hiding this comment.
"PhysicalExpression: a group expression which represents a logical expression..." I think you meant to put "physical expression" instead of "logical expression".
jkosh44
reviewed
Dec 31, 2020
| - `LogicalExpression`: a group expression which represents a logical expression i.e. the logical operator that specifies the general behavior of a node | ||
| - `PhysicalExpression`: a group expression which represents a logical expression i.e. the physical operator that specifies the implementation details of the expression | ||
| - `Rule`: definition of a transformation from one expression to another (may be logical transformation, physical implementation, or rewrite) | ||
| - `Property`: interface to define a physical property, currently of which only sort properties exist |
There was a problem hiding this comment.
I think it would be helpful if we had a couple of sentences explaining what a property is and why we have them.
jkosh44
reviewed
Dec 31, 2020
jkosh44
reviewed
Dec 31, 2020
jkosh44
reviewed
Jan 1, 2021
jkosh44
reviewed
Jan 1, 2021
jkosh44
reviewed
Jan 1, 2021
jkosh44
reviewed
Jan 1, 2021
jkosh44
reviewed
Jan 2, 2021
jkosh44
reviewed
Jan 2, 2021
This was referenced Jan 3, 2021
jkosh44
reviewed
Jan 5, 2021
jkosh44
reviewed
Jan 5, 2021
jkosh44
reviewed
Jan 5, 2021
|
Hey @thepinetree I'm just going to consolidate everything we discussed on slack here so that you have a single place to go to when you return to this PR, let me know if I'm missing anything. I'm not sure which of these belong in this PR, or should be done it later PRs, I guess that's up to you/Will/whoever makes those decisions.
|
thepinetree
added
blocked
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Index Scan + Index Join Limit
Description
Limit clauses are currently not propagated to the
IndexScanPlanNodenor theIndexJoinPlanNodeand as a result, the execution engine can't take advantage of pushing down the limit during operation. Instead, this is done in-post, with aLimitPlanNodedoing so after an index scan is completed.This PR adds functionality for the limit value to be pushed down to index scans, and is used in TPC-C. Limits values will be pushed down to their child
LogicalGetvia transformation rule and converted to values in thePhysicalIndexScanwhich are then set in theIndexScanPlanNode. The PR also moves theOrderByOrderingTypefrom the optimizer to the catalog as a precursor to further changes to involve the sort direction of columns in creating/scanning an index.The final implementation effort is to introduce optional properties to push down ORDER BY sort properties into index scans whenever possible. There are a few alternative possible implementations:
We choose method 1 in this implementation, though the alternative is not difficult to switch to 2 (see PR Index Scan Limit #1031).
Additionally, a guide to optimizer development is included for future developers.
Further work
A description of further work is included in Issue #1421