Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Kernel] Add selection vector create API on ExpressionHandler #2074

Merged
merged 2 commits into from
Sep 20, 2023

Conversation

vkorukanti
Copy link
Collaborator

@vkorukanti vkorukanti commented Sep 18, 2023

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

Part of #2071 (Partition Pruning in Kernel).

Delta kernel-api module creates a selection vector (a ColumnVector boolean data type) that goes along with ColumnarBatch of scan files read from the Delta checkpoint or commit files. It uses the selection vector to select only a subset of rows from the scan file ColumnarBatch. Also at the same time kernel-api module shouldn't be creating ColumnVectors. Instead should rely on the TableClient APIs to create the vectors.

It adds the below API on ExpressionHandler

    /**
     * Create a selection vector, a boolean type {@link ColumnVector}, on top of the range of values
     * given in <i>values</i> array.
     *
     * @param values Array of initial boolean values for the selection vector. The ownership of
     *               this array is with the caller and this method shouldn't depend on it after the
     *               call is complete.
     * @param from   start index of the range, inclusive.
     * @param to     end index of the range, exclusive.
     * @return A {@link ColumnVector} of {@code boolean} type values.
     */
    ColumnVector createSelectionVector(boolean[] values, int from, int to);

This also handles this code review comment.

How was this patch tested?

Added tests.

@vkorukanti vkorukanti merged commit 3648be7 into delta-io:master Sep 20, 2023
5 of 6 checks passed
@vkorukanti vkorukanti deleted the selectionVector branch October 2, 2023 05:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants