[HUDI-7665] Support upgrade downgrade to or from table version 8 #12327

codope · 2024-11-24T17:04:29Z

Change Logs

Migrating table properties including partition fields, key generators, payload type, bootstrap index type. Handling both upgrade and downgrade
Migrating timeline and timeline history to new layout
Full compact the table, to get rid of log files. Both in case of upgrade and downgrade.
Some tests for above.

Impact

Support upgrade downgrade to or from table version 8

Risk level (write none, low medium or high below)

high

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

bvaradar · 2024-11-24T23:42:26Z

...t-common/src/main/java/org/apache/hudi/client/timeline/versioning/v1/TimelineArchiverV1.java

@@ -153,6 +153,21 @@ public int archiveIfRequired(HoodieEngineContext context, boolean acquireLock) t
        txnManager.beginTransaction(Option.empty(), Option.empty());
      }
      List<HoodieInstant> instantsToArchive = getInstantsToArchive().collect(Collectors.toList());
+      return archiveInstants(context, instantsToArchive, false);


We will be locking twice in this code-path. Can we instead introduce a private method which archives a given list of instants and the locking/unlocking managed in the public methods.

I have similiar concerns, the multiple-level falgs for lock control is error-prone, we should avoid that as much as possible, and why we need a lock for upgrade/downgrade then, I don't think there is concurrency here.

Done. I have refactored. Not taking lock here.

...t-common/src/main/java/org/apache/hudi/client/timeline/versioning/v2/TimelineArchiverV2.java

...-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToSevenDowngradeHandler.java

...nt/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java

...di-client-common/src/main/java/org/apache/hudi/table/upgrade/SevenToEightUpgradeHandler.java

...-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToSevenDowngradeHandler.java

danny0405 · 2024-11-25T03:19:26Z

...di-client-common/src/main/java/org/apache/hudi/table/upgrade/SevenToEightUpgradeHandler.java

@@ -58,17 +66,31 @@ public class SevenToEightUpgradeHandler implements UpgradeHandler {
  @Override
  public Map<ConfigProperty, String> upgrade(HoodieWriteConfig config, HoodieEngineContext context,
                                             String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+    config.setValue(HoodieWriteConfig.WRITE_TABLE_VERSION, String.valueOf(HoodieTableVersion.SIX.versionCode()));


why set up this?

so that readers can continue reading at this time

...nt/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java

danny0405 · 2024-11-25T03:34:01Z

...nt/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java

+    };
+    try {
+      HoodieTimelineArchiver archiver = TimelineArchivers.getInstance(TimelineLayoutVersion.LAYOUT_VERSION_2, config, table);
+      archiver.archiveInstants(engineContext, archivedTimeline.getInstants(), true);


I don't think we can read the legady archived instants in one shot. The right manner is reading them in mini-batchs then flush as the new LSM timeline.

And be caution that the legacy archived instants can be large amount, just reading the latest log should be enough for the new file slicing.

read the legacy archived instants as a iterator and flush as a accumulated mini-batch. ArchivedTimelineLoader.loadInstants should be used instead.

I have changed. Now, using LegacyArchivedMetaEntryReader and LSMTimelineWriter directly. Please check latest commit.

danny0405 · 2024-11-25T03:37:55Z

...nt/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java

+        timelineLayoutVersion -> ValidationUtils.checkState(TimelineLayoutVersion.LAYOUT_VERSION_2.equals(timelineLayoutVersion),
+            "Downgrade from LSM timeline is only supported for layout version 2. Given version: " + timelineLayoutVersion));
+    HoodieArchivedTimeline lsmArchivedTimeline = table.getMetaClient().getArchivedTimeline();
+    if (lsmArchivedTimeline.getInstants().isEmpty()) {


read the LSM archived instants as a iterator and flush as a accumulated mini-batch. ArchivedTimelineLoader.loadInstants should be used instead.

revised, please check.

...nt/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java

danny0405 · 2024-11-25T03:43:30Z

hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java

@@ -645,7 +648,9 @@ private static void createTableLayoutOnStorage(StorageConfiguration<?> storageCo
    }

    initializeBootstrapDirsIfNotExists(basePath, storage);
-    HoodieTableConfig.create(storage, metaPathDir, props);
+    if (shouldCreateTableConfig) {


shouldn't we always create the table properties file?

No, I just need to create the timeline/hstory folder without updating the configs.

...nt/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java

address review comments batch instants and use LegacyArchivedMetaEntryReader and LSMTimelineWriter directly Fix archive timeline API and TestSevenToEightUpgradeHandler#testPropertyUpgrade More fixes to TimelineArchiver V2 and V1 Fix the targer archive path for timeline migration[C fix upgrade when auto upgrade is disabled fix upgrade downgrade tests fix testAutoUpgradeFailure Fix the schema mismatch and commit metadata enconding for archived timeline Fix the legacy archival reader for the instant Add a FULL read mode for lsm timeline reader archivePath as an attribute of LSMTimelineWriter and couple other fixes fix archived instant read schema More fix to the LSMTimelineWriter archive path

hudi-bot · 2024-11-27T12:37:00Z

CI report:

8889bf7 UNKNOWN
4aa5210 UNKNOWN
87fd008 Azure: SUCCESS
163c1ba UNKNOWN

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

codope mentioned this pull request Nov 24, 2024

[HUDI-7665] Support rolling upgrade to table version 8 #12250

Closed

4 tasks

github-actions bot added the size:L PR with lines of changes in (300, 1000] label Nov 24, 2024

bvaradar reviewed Nov 25, 2024

View reviewed changes

danny0405 reviewed Nov 25, 2024

View reviewed changes

...-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToSevenDowngradeHandler.java Show resolved Hide resolved

danny0405 reviewed Nov 25, 2024

View reviewed changes

...-client-common/src/main/java/org/apache/hudi/table/upgrade/EightToSevenDowngradeHandler.java Outdated Show resolved Hide resolved

danny0405 reviewed Nov 25, 2024

View reviewed changes

...nt/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java Outdated Show resolved Hide resolved

danny0405 reviewed Nov 25, 2024

View reviewed changes

...nt/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java Outdated Show resolved Hide resolved

danny0405 reviewed Nov 25, 2024

View reviewed changes

...nt/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java Outdated Show resolved Hide resolved

danny0405 reviewed Nov 25, 2024

View reviewed changes

...nt/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngradeUtils.java Show resolved Hide resolved

codope force-pushed the hudi-7665-rolling-upgrade branch from a0e8fb1 to 10c9bbd Compare November 25, 2024 18:33

danny0405 force-pushed the hudi-7665-rolling-upgrade branch from 8889bf7 to c68ad35 Compare November 26, 2024 05:18

github-actions bot added size:XL PR with lines of changes > 1000 and removed size:L PR with lines of changes in (300, 1000] labels Nov 27, 2024

codope force-pushed the hudi-7665-rolling-upgrade branch from 87fd008 to 163c1ba Compare November 27, 2024 12:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-7665] Support upgrade downgrade to or from table version 8 #12327

[HUDI-7665] Support upgrade downgrade to or from table version 8 #12327

codope commented Nov 24, 2024

bvaradar Nov 24, 2024

danny0405 Nov 25, 2024 •

edited

Loading

codope Nov 25, 2024

danny0405 Nov 25, 2024

codope Nov 25, 2024

danny0405 Nov 25, 2024

danny0405 Nov 25, 2024

danny0405 Nov 25, 2024 •

edited

Loading

codope Nov 25, 2024

danny0405 Nov 25, 2024 •

edited

Loading

codope Nov 25, 2024

danny0405 Nov 25, 2024

codope Nov 25, 2024

hudi-bot commented Nov 27, 2024

[HUDI-7665] Support upgrade downgrade to or from table version 8 #12327

Are you sure you want to change the base?

[HUDI-7665] Support upgrade downgrade to or from table version 8 #12327

Conversation

codope commented Nov 24, 2024

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Choose a reason for hiding this comment

danny0405 Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danny0405 Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danny0405 Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hudi-bot commented Nov 27, 2024

CI report:

danny0405 Nov 25, 2024 •

edited

Loading

danny0405 Nov 25, 2024 •

edited

Loading

danny0405 Nov 25, 2024 •

edited

Loading