Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Queries with filter on 2 rel fields of composite index #3035

Conversation

islamaliev
Copy link
Contributor

Relevant issue(s)

Resolves #3032 #2928

Description

Make parallel node perform a real merge instead of copying fields of a fetched doc, which resulted every next doc overwrite previous fields.

@islamaliev islamaliev added bug Something isn't working area/query Related to the query component labels Sep 19, 2024
@islamaliev islamaliev added this to the DefraDB v0.14 milestone Sep 19, 2024
@islamaliev islamaliev requested a review from a team September 19, 2024 15:06
@islamaliev islamaliev self-assigned this Sep 19, 2024
@islamaliev islamaliev changed the title fix: Queries with filter on 2 relation fields of 1 composite index fix: Queries with filter on 2 rel fields of composite index Sep 19, 2024
Copy link
Contributor

@AndrewSisley AndrewSisley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I trust it works, but please explain why the the new code does what it does before merge.

if doc.Fields[i] == nil {
doc.Fields[i] = newFields[i]
} else if newSubDoc, ok := newFields[i].(core.Doc); ok {
doc.Fields[i] = p.mergeDoc(doc.Fields[i].(core.Doc), newSubDoc.Fields)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: Please document why this is needed, it is not terribly clear why the node should be mutating child relations.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, it now justifies the name of method nextMerge.

I'll add a comment.

Copy link
Contributor

@AndrewSisley AndrewSisley Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure the new comment answers what I'm most uncertain about - why does this function recursively mutate child documents, instead of just the top doc? Why and under which circumstances is the mutation of the child documents required? Do please message/call me on discord if you unsure as to what I'm asking.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is no real reason for recursion here. I just though it would cover some future perverted edge-cases.
I made it merge just top fields to keep it simple.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah got it, thanks Islam!

@@ -70,3 +70,106 @@ func TestQueryWithUniqueCompositeIndex_WithFilterOnIndexedRelation_ShouldFilter(

testUtils.ExecuteTestCase(t, test)
}

func TestQueryWithUniqueCompositeIndex_WithIndexComprising2RelationsAndFilterOnIt_ShouldFilter(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: It might be highlighting with a comment why this test is important to have, for example compared to just for composite scalar tests.

Please also document the significance of including two id fields, and one object field in the index. Perhaps highlighting why a test with two object fields in the index (e.g. includes: [{name: "owner"}, {name: "manufacturer"} is not required.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test case is actually a bit simpler. I wanted to update it, but forgot. Thanks for reminding.

I will add some comments.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Islam, looks good!

Copy link

codecov bot commented Sep 19, 2024

Codecov Report

Attention: Patch coverage is 97.61905% with 1 line in your changes missing coverage. Please review.

Project coverage is 79.35%. Comparing base (926c334) to head (aa3f3fe).
Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
internal/planner/type_join.go 95.65% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #3035      +/-   ##
===========================================
- Coverage    79.35%   79.35%   -0.00%     
===========================================
  Files          333      333              
  Lines        25837    25841       +4     
===========================================
+ Hits         20501    20504       +3     
- Misses        3866     3869       +3     
+ Partials      1470     1468       -2     
Flag Coverage Δ
all-tests 79.35% <97.62%> (-<0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
internal/planner/multi.go 84.07% <100.00%> (+0.29%) ⬆️
internal/planner/scan.go 90.86% <100.00%> (-0.67%) ⬇️
internal/planner/type_join.go 81.73% <95.65%> (-0.27%) ⬇️

... and 12 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 926c334...aa3f3fe. Read the comment docs.

@islamaliev islamaliev force-pushed the fix/unique-index-return-all-children branch from 4d75ba0 to 9519e28 Compare September 19, 2024 16:50
Copy link
Contributor

@AndrewSisley AndrewSisley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks Islam - please try and tweak the documentation for mergeDoc a little bit before merge though.

@islamaliev islamaliev merged commit 5754d7d into sourcenetwork:develop Sep 19, 2024
41 of 43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/query Related to the query component bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Secondary indexes on relations cause different documents to be returned depending on joins included
2 participants