[Kernel] Add partition pruning related utility methods #2098
Conversation
vkorukanti
force-pushed
the
partitionUtils
branch
2 times, most recently
from
fbe875f
to
5b46cd0
Compare
| Set<String> partitionColNames) { | ||
| for (Expression child : children) { | ||
| if (child instanceof Column) { | ||
| String[] names = ((Column) child).getNames(); |
There was a problem hiding this comment.
hm. i'm just noticing this getNames API. IMO getNames implies that the Column has multiple names, no? that seems odd. nestedName would be better?
There was a problem hiding this comment.
Spark has filedNames. nameParts is another option. nestedName could imply it is just for nested columns.
There was a problem hiding this comment.
Doesn't need to be done in this PR. We can create an issue and assign it to the 3.1 milestone.
kernel/kernel-api/src/main/java/io/delta/kernel/internal/util/PartitionUtils.java
Show resolved
Hide resolved
kernel/kernel-api/src/main/java/io/delta/kernel/internal/util/PartitionUtils.java
Outdated
Show resolved
Hide resolved
kernel/kernel-api/src/main/java/io/delta/kernel/internal/util/PartitionUtils.java
Show resolved
Hide resolved
kernel/kernel-api/src/test/scala/io/delta/kernel/internal/util/PartitionUtilsSuite.scala
Outdated
Show resolved
Hide resolved
| val partitionTestCases = Map[Predicate, (String, String)]( | ||
| // single predicate on a data column | ||
| predicate("=", col("data1"), ofInt(12)) -> | ||
| ("ALWAYS_TRUE()", "(column(`data1`) = 12)"), |
There was a problem hiding this comment.
nit: probably too late, but it should ideally just be
`column(data1)` instead of
column(`data1`)right?
i.e. we should only use back ticks for columns with .s in them?
column(a.b.`col.with.dot`.e)
Isn't this what spark does?
There was a problem hiding this comment.
It is just a toString method. If I recall correctly Delta-Spark does the same (add `` for every column). I can make it just adding backtick for names containing ., not sure if it is worth it.
There was a problem hiding this comment.
Cool. I just want to do what Spark does.
kernel/kernel-api/src/test/scala/io/delta/kernel/internal/util/PartitionUtilsSuite.scala
Outdated
Show resolved
Hide resolved
vkorukanti
force-pushed
the
partitionUtils
branch
from
5b46cd0
to
6440261
Compare
Which Delta project/connector is this regarding?
Description
Part of #2071 (Partition Pruning in Kernel). This PR adds the following utility methods:
Predicategiven to theScanBuilder.withFilterinto data column and partition column predicatesPredicateto refer to the columns in the scan file columnar batch with the appropriate partition value deserialization expressions applied.How was this patch tested?
Added UTs