Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correct protocol downgrade error when some table features are enabled #2062

Closed
wants to merge 1 commit into from

Conversation

xupefei
Copy link
Contributor

@xupefei xupefei commented Sep 15, 2023

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

This PR improves the logic of deciding protocol versions in ALTER TABLE commands. Specifically, when a query specifies a lower protocol version along with a table prop that enables a table feature, we will try to "correct" the lower protocol version to a higher one that supports the said table feature. Some examples:

-- table has Protocol(2, 2)
ALTER TABLE table SET TBLPROPERTIES (
	delta.minWriterVersion = '1',
	delta.enableChangeDataFeed = 'true'
)
-- before: cannot downgrade from Protocol(2, 2) to Protocol(2, 1)
-- after: table now have Protocol(2, 4)
-- table has Protocol(2, 2)
ALTER TABLE table SET TBLPROPERTIES (
	delta.minReaderVersion = '1',
	delta.minWriterVersion = '1',
	delta.enableChangeDataFeed = 'true'
)
-- before: cannot downgrade from Protocol(2, 2) to Protocol(1, 1)
-- after: cannot downgrade from Protocol(2, 2) to Protocol(1, 4)
-- table has Protocol(2, 2)
ALTER TABLE table SET TBLPROPERTIES (
	delta.minReaderVersion = '1',
	delta.minWriterVersion = '1',
	delta.enableDeletionVectors = 'true'
)
-- before: cannot downgrade from Protocol(2, 2) to Protocol(1, 1)
-- after: table now have Protocol(3, 7)

How was this patch tested?

New tests.

Does this PR introduce any user-facing changes?

Yes. See the first section.

@xupefei xupefei changed the title . Correct protocol downgrade when some table features are enabled Sep 15, 2023
@xupefei xupefei changed the title Correct protocol downgrade when some table features are enabled Correct protocol downgrade error when some table features are enabled Sep 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants