Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-3070: Add Variant logical type annotation to parquet-java #3072

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

aihuaxu
Copy link

@aihuaxu aihuaxu commented Nov 22, 2024

Rationale for this change

This is to add Variant logical type in parquet-java to be used by dependent projects.

What changes are included in this PR?

Variant logical type is added to LogicalTypeAnnotation.

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes. Variant logical type is available.

Closes #3070

@aihuaxu
Copy link
Author

aihuaxu commented Nov 22, 2024

@Fokko, @wgtmac Can you help check if this is right implementation? Thanks.

@wgtmac
Copy link
Member

wgtmac commented Nov 22, 2024

Do we need to support writing and reading variant data?

@aihuaxu
Copy link
Author

aihuaxu commented Nov 22, 2024

The Variant reading and writing are getting implemented in Iceberg and/or the engines themselves. I think later we can think of pulling the implementation to Parquet if needed.

@emkornfield
Copy link
Contributor

The Variant reading and writing are getting implemented in Iceberg and/or the engines themselves. I think later we can think of pulling the implementation to Parquet if needed.

I think this is problematic if the spec lives in parquet and doesn't have a complete implementation per previously agreed upon guidelines for new parquet features. This probably warrants a discussion on the mailing list. CC @julienledem @rdblue @RussellSpitzer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Variant Logical Type
3 participants