VerifyException during ordering calculation for IntersectionOnValuesPlan

[alecgrieser]

I ran into a planner VerifyException in a test query when trying to determine the ordering property on an intersection on values plan. I was able to reproduce it fairly reliably in the FDBSimpleQueryGraphTest with a test for a query like:

SELECT R.rest_no
   FROM RestaurantRecord AS R
   WHERE R.name IN $nameList
       AND EXISTS (SELECT t.value FROM R.tags AS t WHERE t.value = $tagValue)

This error seems to only show up when there's an index on concat(field("name"), field("tags", FanOut).nest("value")). I believe it's trying to evaluate the query by issuing one index scan of the index for each entry in nameList and then merge them together using an intersection plan, but there are problems getting the intersection plan to work.

        final Index index = new Index("Restaurant$name-tagValue", Key.Expressions.concat(
                Key.Expressions.field("name"),
                Key.Expressions.field("tags").nest(Key.Expressions.field("values", KeyExpression.FanType.FanOut).nest("value"))
        ));
        RecordMetaDataHook hook = metaDataBuilder -> metaDataBuilder.addIndex("RestaurantRecord", index);
        CascadesPlanner cascadesPlanner = setUpWithNullableArray(hook);

        final var nameValueParam = "name";
        final var tagValueParam = "t";
        final var plan = planGraph(
                () -> {
                    var qun = fullTypeScan(cascadesPlanner.getRecordMetaData(), "RestaurantRecord");

                    final var explodeTagsQun = Quantifier.forEach(Reference.of(new ExplodeExpression(FieldValue.ofFieldName(qun.getFlowedObjectValue(), "tags"))));
                    final var existentialQun = Quantifier.existential(Reference.of(GraphExpansion.builder()
                            .addQuantifier(explodeTagsQun)
                            .addResultColumn(projectColumn(explodeTagsQun.getFlowedObjectValue(), "value"))
                            .addPredicate(new ValuePredicate(FieldValue.ofFieldName(explodeTagsQun.getFlowedObjectValue(), "value"),
                                    new Comparisons.ParameterComparison(Comparisons.Type.EQUALS, tagValueParam)))
                            .build()
                            .buildSelect()));

                    qun = Quantifier.forEach(Reference.of(GraphExpansion.builder()
                            .addQuantifier(qun)
                            .addQuantifier(existentialQun)
                            .addPredicate(new ValuePredicate(FieldValue.ofFieldName(qun.getFlowedObjectValue(), "name"), new Comparisons.ParameterComparison(Comparisons.Type.IN, nameValueParam)))
                            .addPredicate(new ExistsPredicate(existentialQun.getAlias()))
                            .addResultColumn(projectColumn(qun.getFlowedObjectValue(), "rest_no"))
                            .build()
                            .buildSelect()));
                    return Reference.of(LogicalSortExpression.unsorted(qun));
                });

Here's a stack trace from trying to plan that query in the test:

com.google.common.base.VerifyException
	at com.google.common.base.Verify.verify(Verify.java:102)
	at com.apple.foundationdb.record.query.plan.cascades.Ordering$SetOperationsOrdering.lambda$applyComparisonKey$2(Ordering.java:1335)
	at com.apple.foundationdb.record.query.plan.cascades.debug.Debugger.lambda$sanityCheck$0(Debugger.java:118)
	at com.apple.foundationdb.record.query.plan.cascades.debug.Debugger.withDebugger(Debugger.java:107)
	at com.apple.foundationdb.record.query.plan.cascades.debug.Debugger.sanityCheck(Debugger.java:116)
	at com.apple.foundationdb.record.query.plan.cascades.Ordering$SetOperationsOrdering.applyComparisonKey(Ordering.java:1335)
	at com.apple.foundationdb.record.query.plan.cascades.properties.OrderingProperty$OrderingVisitor.deriveForDistinctSetOperationFromOrderings(OrderingProperty.java:663)
	at com.apple.foundationdb.record.query.plan.cascades.properties.OrderingProperty$OrderingVisitor.visitIntersectionOnValuesPlan(OrderingProperty.java:296)
        at com.apple.foundationdb.record.query.plan.cascades.properties.OrderingProperty$OrderingVisitor.visitIntersectionOnValuesPlan(OrderingProperty.java:116)
	at com.apple.foundationdb.record.query.plan.plans.RecordQueryPlanVisitor.lambda$static$13(RecordQueryPlanVisitor.java:24)
	at com.apple.foundationdb.record.query.plan.plans.RecordQueryPlanVisitor.visit(RecordQueryPlanVisitor.java:168)
	at com.apple.foundationdb.record.query.plan.cascades.PropertiesMap.computePropertyValue(PropertiesMap.java:168)
	at com.apple.foundationdb.record.query.plan.cascades.PropertiesMap.update(PropertiesMap.java:110)
	at com.apple.foundationdb.record.query.plan.cascades.PropertiesMap.getPlanPartitions(PropertiesMap.java:199)
	at com.apple.foundationdb.record.query.plan.cascades.Reference.getPlanPartitions(Reference.java:383)
	at com.apple.foundationdb.record.query.plan.cascades.matching.structure.Extractor.lambda$new$0(Extractor.java:40)
	at com.apple.foundationdb.record.query.plan.cascades.matching.structure.Extractor.unapply(Extractor.java:51)
	at com.apple.foundationdb.record.query.plan.cascades.matching.structure.TypedMatcherWithExtractAndDownstream.lambda$bindMatchesSafely$0(TypedMatcherWithExtractAndDownstream.java:67)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
	at java.base/java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
	at java.base/java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at com.apple.foundationdb.record.query.plan.cascades.matching.structure.ListMatcher.bindMatchesSafely(ListMatcher.java:66)
	at com.apple.foundationdb.record.query.plan.cascades.matching.structure.ListMatcher.bindMatchesSafely(ListMatcher.java:44)
	at com.apple.foundationdb.record.query.plan.cascades.matching.structure.BindingMatcher.bindMatches(BindingMatcher.java:94)
	at com.apple.foundationdb.record.query.plan.cascades.matching.structure.TypedMatcherWithExtractAndDownstream.lambda$bindMatchesSafely$0(TypedMatcherWithExtractAndDownstream.java:67)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
	at java.base/java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
	at java.base/java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
	at java.base/java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
	at java.base/java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
	at java.base/java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at com.apple.foundationdb.record.query.plan.cascades.CascadesPlanner$AbstractTransform.execute(CascadesPlanner.java:862)
	at com.apple.foundationdb.record.query.plan.cascades.CascadesPlanner.planPartial(CascadesPlanner.java:405)
	at com.apple.foundationdb.record.query.plan.cascades.CascadesPlanner.planGraph(CascadesPlanner.java:344)
	at com.apple.foundationdb.record.provider.foundationdb.query.FDBRecordStoreQueryTestBase.planGraph(FDBRecordStoreQueryTestBase.java:644)
	at com.apple.foundationdb.record.provider.foundationdb.query.FDBRecordStoreQueryTestBase.planGraph(FDBRecordStoreQueryTestBase.java:631)