Range Query Optimization (For sequential Vindex types)

[c-r-dev]

Description

Queries that specify values in a range (for vindexed column) are sent to all shards, instead of just the shards which contain the values in the range.

• For primary vindexs which are sequential and not random, a BINARY vindex for example, optimized range queries at vtgate to be sent to only shards included between the first and last keyspaceid values of the range.

Related Issue(s)

Fixes #9808

Checklist

• "Backport to:" labels have been added if this change should be back-ported to release branches
• If this change is to be back-ported to previous releases, a justification is included in the PR description
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on CI?
• Documentation was added or is not required

Deployment Notes

Made the change for sequential Vindexes ( binary and numeric)

Tests Performed

• Added unit tests module level

• Included sample schema with two columns (c1 and c2) , c1 is vindexed (type binday) and c2 is not vindexed.

When range queried on column c1 (vindex column) :

bin % ./vtexplain --shards 4 --vschema-file ../examples/vtexplain/sample_vschema.json --schema-file ../examples/vtexplain/sample_schema.sql --sql "select * from t1 where c1 between 1 and 10"

----------------------------------------------------------------------
select * from t1 where c1 between 1 and 10

1 ks1/-40: select * from t1 where c1 between 1 and 10 limit 10001

----------------------------------------------------------------------
<nil>%                                      

When range queried on column c2 (non vindex column) :

bin % ./vtexplain --shards 4 --vschema-file ../examples/vtexplain/sample_vschema.json --schema-file ../examples/vtexplain/sample_schema.sql --sql "select * from t1 where c2 between 1 and 10"

----------------------------------------------------------------------
select * from t1 where c2 between 1 and 10

1 ks1/-40: select * from t1 where c2 between 1 and 10 limit 10001
1 ks1/40-80: select * from t1 where c2 between 1 and 10 limit 10001
1 ks1/80-c0: select * from t1 where c2 between 1 and 10 limit 10001
1 ks1/c0-: select * from t1 where c2 between 1 and 10 limit 10001

----------------------------------------------------------------------
<nil>%                                                                                                                                                                                                           

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[deepthi]

I unchecked this, because CI has not yet run

Did the new or modified tests pass consistently locally and on CI?

[c-r-dev]

I unchecked this, because CI has not yet run

Did the new or modified tests pass consistently locally and on CI?

thanks, confirm - modified tests passed consistently locally ( i have run module level unit tests )

vitess.io/vitess/go/vt/key
vitess.io/vitess/go/vt/vtgate/engine
vitess.io/vitess/go/vt/vtgate/vindexes
vitess.io/vitess/go/vt/vtgate/planbuilder/operators

Didn't get CI to run locally end to end. Currently working on the CI setup.

[systay]

Thank you so much for your contribution!
This looks really good from my POV.

I would love to see a plan test and and one end-to-end test as well. Not sure where the end-to-end test should live. @harshit-gangal probably knows.

[harshit-gangal]

We should add plan test that uses Between for Sequential Vindex and Non-Sequential Vindex type

It can be added to go/vt/vtgate/planbuilder/testdata/select_cases.json

[c-r-dev]

thanks @systay and @harshit-gangal for reviewing and sharing feedback, let me incorporate these and get back. Will reach out in case of questions.

[c-r-dev]

We should add plan test that uses Between for Sequential Vindex and Non-Sequential Vindex type

It can be added to go/vt/vtgate/planbuilder/testdata/select_cases.json

Thanks for the suggestion and pointing me to sample , i have added new test cases with e8593a8

[c-r-dev]

thanks @systay , will check these and make next set of changes.

[codecov]

Codecov Report

Attention: Patch coverage is 44.70588% with 47 lines in your changes missing coverage. Please review.

Project coverage is 67.63%. Comparing base (56b5c2a) to head (01007a5).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
go/vt/vtgate/engine/routing.go	0.00%	30 Missing ⚠️
go/vt/vtgate/vindexes/numeric.go	0.00%	11 Missing ⚠️
go/vt/vtgate/vindexes/binary.go	63.63%	4 Missing ⚠️
...vt/vtgate/planbuilder/operators/sharded_routing.go	93.54%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #17342      +/-   ##
==========================================
+ Coverage   67.60%   67.63%   +0.03%     
==========================================
  Files        1581     1581              
  Lines      253945   253860      -85     
==========================================
+ Hits       171670   171708      +38     
+ Misses      82275    82152     -123     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[harshit-gangal]

Changes looks good.
For end to end test. We recently added a way to test the plan tests to also run as e2e test

Look here https://github.com/vitessio/vitess/blob/main/go/test/endtoend/vtgate/plan_tests/plan_e2e_test.go
and the PR that added it #17117

You would have to add filter_cases.json to the e2e test.

[c-r-dev]

Changes looks good. For end to end test. We recently added a way to test the plan tests to also run as e2e test

Look here https://github.com/vitessio/vitess/blob/main/go/test/endtoend/vtgate/plan_tests/plan_e2e_test.go and the PR that added it #17117

You would have to add filter_cases.json to the e2e test.

thanks @harshit-gangal , will pull in the changes from #17117 and add test case for filer_case.json. Will reach out in case of any questions.

[c-r-dev]

Changes looks good. For end to end test. We recently added a way to test the plan tests to also run as e2e test
Look here https://github.com/vitessio/vitess/blob/main/go/test/endtoend/vtgate/plan_tests/plan_e2e_test.go and the PR that added it #17117
You would have to add filter_cases.json to the e2e test.

thanks @harshit-gangal , will pull in the changes from #17117 and add test case for filter_case.json. Will reach out in case of any questions.

Thanks , added end-to-end test case for filter_case.json with f4a578c. Please help to review if skip_e2e is tagged correctly ( kept it similar to select_case.json - join cases (no sample data in table) and known error cases )

[c-r-dev]

thanks @harshit-gangal for reviewing , will incorporate feedback and share and updated version for review.

[c-r-dev]

Thank you so much for your contribution! This looks really good from my POV.

I would love to see a plan test and and one end-to-end test as well. Not sure where the end-to-end test should live. @harshit-gangal probably knows.

@systay , thanks for the feedback , have worked with @harshit-gangal and got end-to-end setup for filter_case.

Can you please help to review when you get a chance.

[c-r-dev]

thanks @systay for helping to review, have incorporated feedback. Please let me know how it looks.