feat: Inline array filters #3028

nasdf · 2024-09-18T00:02:55Z

Relevant issue(s)

Resolves #2857

Description

This PR adds three new filter types for inline array types: _all, _any, and _none.

Tasks

I made sure the code is well commented, particularly hard-to-understand areas.
I made sure the repository-held documentation is changed accordingly.
I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, ...

How has this been tested?

Added integration tests

Specify the platform(s) on which this was tested:

MacOS

codecov · 2024-09-18T00:22:16Z

Codecov Report

Attention: Patch coverage is 94.40559% with 16 lines in your changes missing coverage. Please review.

Project coverage is 79.46%. Comparing base (cddc6d6) to head (307ca23).
Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
internal/connor/all.go	75.00%	5 Missing and 2 partials ⚠️
internal/connor/any.go	75.00%	5 Missing and 2 partials ⚠️
internal/connor/none.go	60.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3028      +/-   ##
===========================================
+ Coverage    79.45%   79.46%   +0.01%     
===========================================
  Files          333      336       +3     
  Lines        25840    26075     +235     
===========================================
+ Hits         20530    20720     +190     
- Misses        3853     3883      +30     
- Partials      1457     1472      +15

Flag	Coverage Δ
all-tests	`79.46% <94.41%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
internal/connor/and.go	`54.55% <ø> (ø)`
internal/connor/connor.go	`100.00% <100.00%> (ø)`
internal/connor/eq.go	`94.87% <100.00%> (+2.71%)`	⬆️
internal/planner/mapper/mapper.go	`88.83% <100.00%> (ø)`
internal/request/graphql/schema/generate.go	`88.05% <100.00%> (+0.10%)`	⬆️
internal/request/graphql/schema/manager.go	`98.67% <100.00%> (+0.23%)`	⬆️
internal/request/graphql/schema/types/base.go	`100.00% <100.00%> (ø)`
internal/connor/none.go	`60.00% <60.00%> (ø)`
internal/connor/all.go	`75.00% <75.00%> (ø)`
internal/connor/any.go	`75.00% <75.00%> (ø)`

... and 15 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cddc6d6...307ca23. Read the comment docs.

fredcarle

LGTM! Very nice and thanks for all the tests.

islamaliev

Looks good. I just requested few changes.

islamaliev · 2024-09-18T06:03:48Z

internal/connor/all.go

+// matching if all of them match.
+func all(condition, data any) (bool, error) {
+	switch t := data.(type) {
+	case []string:


todo: I think it's better to use reflection here to avoid repetitive code:

func all(condition, data any) { val := reflect.ValueOf(v) if val.Kind() != reflect.Slice { return false, client.NewErrUnhandledType("data", data) } for i := 0; i < val.Len(); i++ { m, err := eq(condition, val.Index(i).Interface()) if err != nil { return false, err } else if !m { return false, nil } } } return true, nil }

Or at least use generic:

func allInArr[T any](slice []T, condition any) (bool, error) { for _, item := range slice { m, err := eq(condition, item) if err != nil { return false, err } if !m { return false, nil } } return true, nil } func all(condition, data any) (bool, error) { switch t := data.(type) { case []string: return allInArr(t, condition) case []immutable.Option[string]: return allInArr(t, condition) case []int64: return allInArr(t, condition) // ...

This will also improve test coverage.

reflect is a bit slow and not fully supported in wasm so lets try to avoid it for now. I like the generic idea though.

100% agree with Fred, we should avoid reflect, especially in a performance sensitive area like this. Extracting the shared logic to a generic func would be good though.

islamaliev · 2024-09-18T06:04:39Z

internal/connor/any.go

+// matching if any of them match.
+func anyOp(condition, data any) (bool, error) {
+	switch t := data.(type) {
+	case []string:


todo: please use the same approach as with all

islamaliev · 2024-09-18T06:06:35Z

internal/connor/eq.go

+		if arr.HasValue() {
+			data = arr.Value()
+		} else {
+			data = nil


suggestion: this could also be turned into generic code

islamaliev · 2024-09-18T06:14:18Z

internal/request/graphql/schema/types/base.go

+		Fields: gql.InputObjectConfigFieldMap{
+			"_any": &gql.InputObjectFieldConfig{
+				Description: anyOperatorDescription,
+				Type:        op,
+			},
+			"_all": &gql.InputObjectFieldConfig{
+				Description: allOperatorDescription,
+				Type:        op,
+			},
+			"_none": &gql.InputObjectFieldConfig{
+				Description: noneOperatorDescription,
+				Type:        op,
+			},
+		},


suggestion: this can be extracted into a function and used everywhere

I have mixed feelings about the extraction of the full InputObjectFieldConfig blocks to a shared func (it couples them together, and hides what each InputObject contains, yet the enforcement of consistency is nice).

My preference is to keep them as is because it makes modifying each individual type easier.

islamaliev · 2024-09-18T06:19:39Z

tests/integration/query/inline_array/with_filter_all_test.go

+	testUtils "github.com/sourcenetwork/defradb/tests/integration"
+)
+
+func TestQueryInlineStringArrayWithAllFilter(t *testing.T) {


suggestion: we try to stick to the following test naming convention:
<Category>_<Condition|Prerequisite>_<Result|Expectation>
In this case it could be
TestQueryInlineStringArray_WithAllFilter_Succeeds

nitpick: I did think we agreed to not force that suggestion upon developers though, especially the last block.

I agree that that last part is optional

islamaliev · 2024-09-18T06:26:41Z

tests/integration/query/inline_array/with_filter_all_test.go

+					Users(filter: {testScores: {_all: {_lt: 70}}}) {
+						name
+					}


suggestion: would be also great to see a reassurance (in form of another test) that this new op works with compound conditions. For example:
filter: {testScores: {_all: {_and: [{_lt: 70}, {_gte: 0}]}}}
And to do the same with other new ops.

I've added two compound tests.

AndrewSisley · 2024-09-18T13:01:53Z

internal/request/graphql/schema/manager.go

+	notNullBooleanOpBlock := schemaTypes.NotNullBooleanOperatorBlock()
+	notNullStringOpBlock := schemaTypes.NotNullStringOperatorBlock()
+	notNullJSONOpBlock := schemaTypes.NotNullJSONOperatorBlock(jsonScalarType)
+	notNullBlobOpBlock := schemaTypes.NotNullBlobOperatorBlock(blobScalarType)


question: I'm curious - why did you pull the existing stuff out of the return and into temp vars? And then add new func calls within the return?

It made sense to me organizationally. Also the temp vars are used as inputs to the calls in the return.

AndrewSisley · 2024-09-18T13:06:47Z

internal/request/graphql/schema/types/base.go

+		Name:        "BooleanListOperatorBlock",
+		Description: "These are the set of filter operators available for use when filtering on [Boolean] values.",
+		Fields: gql.InputObjectConfigFieldMap{
+			"_any": &gql.InputObjectFieldConfig{


suggestion: Independent of Islam's suggestion, it might be nice to extract the names to consts in the client/request package and then also reference them in the connor package as well as here.

This is out of scope for this PR imo, but I will create an issue for this.

islamaliev

LGTM!

shahzadlone

LGTM

islamaliev · 2024-10-16T05:13:02Z

Bug bash results: all good

add inline array filter types

fe668a2

nasdf added feature New feature or request area/query Related to the query component labels Sep 18, 2024

nasdf added this to the DefraDB v0.14 milestone Sep 18, 2024

nasdf self-assigned this Sep 18, 2024

nasdf requested a review from a team September 18, 2024 00:03

fredcarle approved these changes Sep 18, 2024

View reviewed changes

islamaliev requested changes Sep 18, 2024

View reviewed changes

AndrewSisley reviewed Sep 18, 2024

View reviewed changes

nasdf and others added 3 commits September 18, 2024 10:37

simplify with generics

9c1b059

add compound tests. update test names

4b4c822

Merge branch 'develop' into nasdf/feat/inline-array-filter

a467f1e

nasdf requested a review from islamaliev September 18, 2024 17:41

nasdf and others added 2 commits September 18, 2024 16:01

fix ordering of filter test results

dfbf78a

Merge branch 'develop' into nasdf/feat/inline-array-filter

670f3b2

islamaliev approved these changes Sep 19, 2024

View reviewed changes

shahzadlone approved these changes Sep 19, 2024

View reviewed changes

nasdf and others added 3 commits September 19, 2024 13:13

Merge branch 'develop' into nasdf/feat/inline-array-filter

17b7482

revert schema fixture change

5727a12

Merge branch 'develop' into nasdf/feat/inline-array-filter

307ca23

nasdf merged commit 55e56a5 into sourcenetwork:develop Sep 20, 2024
42 of 43 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Inline array filters #3028

feat: Inline array filters #3028

nasdf commented Sep 18, 2024

codecov bot commented Sep 18, 2024 •

edited

Loading

fredcarle left a comment

islamaliev left a comment •

edited

Loading

islamaliev Sep 18, 2024

fredcarle Sep 18, 2024 •

edited

Loading

AndrewSisley Sep 18, 2024

nasdf Sep 18, 2024

islamaliev Sep 18, 2024

nasdf Sep 18, 2024

islamaliev Sep 18, 2024

nasdf Sep 18, 2024

islamaliev Sep 18, 2024

AndrewSisley Sep 18, 2024

nasdf Sep 18, 2024

islamaliev Sep 18, 2024

AndrewSisley Sep 18, 2024

islamaliev Sep 18, 2024

nasdf Sep 18, 2024

islamaliev Sep 18, 2024 •

edited

Loading

nasdf Sep 18, 2024

AndrewSisley Sep 18, 2024

nasdf Sep 18, 2024

AndrewSisley Sep 18, 2024

nasdf Sep 18, 2024

islamaliev left a comment

shahzadlone left a comment

islamaliev commented Oct 16, 2024

feat: Inline array filters #3028

feat: Inline array filters #3028

Conversation

nasdf commented Sep 18, 2024

Relevant issue(s)

Description

Tasks

How has this been tested?

codecov bot commented Sep 18, 2024 • edited Loading

Codecov Report

fredcarle left a comment

Choose a reason for hiding this comment

islamaliev left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fredcarle Sep 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

islamaliev Sep 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

islamaliev left a comment

Choose a reason for hiding this comment

shahzadlone left a comment

Choose a reason for hiding this comment

islamaliev commented Oct 16, 2024

codecov bot commented Sep 18, 2024 •

edited

Loading

islamaliev left a comment •

edited

Loading

fredcarle Sep 18, 2024 •

edited

Loading

islamaliev Sep 18, 2024 •

edited

Loading