[controller] Fix mismatch between hybrid version partition count and real-time partition count #1338

sushantmane · 2024-11-22T18:22:58Z

Ensure real-time topic partition count matches hybrid version partition count

This fix addresses an issue where the real-time topic partition count did not align with the hybrid version
partition count, causing errors during hybrid store ingestion. The issue occurred in the following scenario:

Create a store with 1 partition.
Perform a batch push, creating a batch version with 1 partition.
Update the store to 3 partitions and convert it to a hybrid store.
Start real-time writes using push type STREAM.
Perform a full push to create a hybrid version with 3 partitions. This push fails because, after the topic
switch, real-time consumers cannot find partitions 2 and 3 due to the real-time topic having only 1 partition.

Root Cause:

In step 4, if the real-time topic did not exist, it was created with a partition count derived from the largest
existing version (batch version with 1 partition), which lead to mismatch.

Solution in this PR:

STREAM push type is now disallowed if there is no online hybrid version.
If an online hybrid version exists, it ensures the real-time topic partition count matches the hybrid version
partition count.
The requestTopicForPushing method no longer creates a real-time topic if it does not already exist.

Misc:

Refactor CreationVersion::requestTopicForPushing to make it easy for unit testing
Added similar checks for incremental push job type
Fix flaky blob transfer flaky tests
Stop creating real-time topics for region system stores like meta and PS3 in parent region
Logging changes around TS to make it easy to search

How was this PR tested?

WIP

Does this PR introduce any user-facing changes?

No. You can skip the rest of this section.
Yes. Make sure to explain your proposed changes and call out the behavior change.

sushantmane · 2024-11-22T19:51:52Z

Moving to draft as I would like to move topic creation in addVersion and see how that works out

...ient/src/main/java/com/linkedin/davinci/kafka/consumer/LeaderFollowerStoreIngestionTask.java

...enice-common/src/main/java/com/linkedin/venice/controllerapi/RequestTopicForPushRequest.java

lusong64 · 2024-12-03T00:09:43Z

...ces/venice-controller/src/main/java/com/linkedin/venice/controller/server/CreateVersion.java

+
+  private static void verifyPartitioner(PartitionerConfig storePartitionerConfig, Set<String> partitionersFromRequest) {
+    // If partitioners are provided, check if the store partitioner is in the list
+    if (partitionersFromRequest != null && !partitionersFromRequest.isEmpty()


is the isEmpty test here necessary? If a set is empty, your third test contains(xXX) will return false.

Yes, the partitionersFromRequest.isEmpty() check is necessary. The intent of the method is to verify if the optional partitioner list provided by the user contains the partitioner from the store’s configuration. If we do not explicitly check for isEmpty, and the user provides an empty set, we would inadvertently throw an exception. This is not the expected behavior because an empty set implies that no specific partitioners were requested, and we should simply fall back to the store’s partitioner without raising an error.

...ces/venice-controller/src/main/java/com/linkedin/venice/controller/server/CreateVersion.java

arjun4084346 · 2024-12-05T23:31:57Z

I agree with the solution "STREAM push type is now disallowed if there is no online hybrid version." However, I think, instead of returning error, we should create a new version (that would be of right partition count) and then let the push start.
If we do not auto create the version, we can at least tell the user to create a version, if we find store's partition count different from the VT's partition count.

If we look carefully, changing partition count without creating a version (even for batch stores) is similar to changing partition for a hybrid store; even for the batch stores, because batch stores can be changed to hybrid

I am skeptical about "The requestTopicForPushing method no longer creates a real-time topic if it does not already exist."; I believe in most of the cases, this is a benign operation and we should not prevent creation of the RT.

sushantmane · 2024-12-06T17:52:29Z

I agree with the solution "STREAM push type is now disallowed if there is no online hybrid version." However, I think, instead of returning error, we should create a new version (that would be of right partition count) and then let the push start. If we do not auto create the version, we can at least tell the user to create a version, if we find store's partition count different from the VT's partition count.

If we look carefully, changing partition count without creating a version (even for batch stores) is similar to changing partition for a hybrid store; even for the batch stores, because batch stores can be changed to hybrid

I am skeptical about "The requestTopicForPushing method no longer creates a real-time topic if it does not already exist."; I believe in most of the cases, this is a benign operation and we should not prevent creation of the RT.

we should create a new version (that would be of right partition count)

I think this is not safe in some cases, for example, user wants to create a new version but keep old data.

If we look carefully, changing partition count without creating a version (even for batch stores) is similar to changing partition for a hybrid store; even for the batch stores, because batch stores can be changed to hybrid

I'm don't really follow the rest of the comment. Could you please elaborate? Thanks

changing partition for a hybrid store

We do not allow this as of now

I believe in most cases, this is a harmless operation, and we should not prevent the creation of the RT.

I disagree that this is a harmless operation. The case mentioned in the commit message is one example why this is not a good idea. Moreover, we should tie hybrid version materialization to RT creation for better alignment and not let RT creation happen at every other place IMHO

…rtition count This fix addresses an issue where the real-time topic partition count did not align with the hybrid version partition count, causing errors during hybrid store operations. The issue occurred in the following scenario: 1. Create a store with 1 partition. 2. Perform a batch push, creating a batch version with 1 partition. 3. Update the store to 3 partitions and convert it to a hybrid store. 4. Start real-time writes using push type STREAM. 5. Perform a full push to create a hybrid version with 3 partitions. This push fails because, after the topic switch, real-time consumers cannot find partitions 2 and 3 due to the real-time topic having only 1 partition. Root Cause: - In step 4, if the real-time topic did not exist, it was created with a partition count derived from the largest existing version (batch version with 1 partition), leading to a mismatch. Solution: - STREAM push type is now disallowed if there is no online hybrid version. - If an online hybrid version exists, it ensures the real-time topic partition count matches the hybrid version partition count. - The `requestTopicForPushing` method no longer creates a real-time topic if it does not already exist. Move real-time topic creation logic in addVersion enable participant message store and revert HB interval Fix tests in 1430 Fix test Fix tests Fix cc tests revert log4j2 Disable topic creation in RT topic switcher Fix flakies Do not call getRealTimeTopic fix tests

arjun4084346 · 2024-12-19T21:31:46Z

internal/venice-common/src/main/java/com/linkedin/venice/pubsub/PubSubConstants.java

@@ -65,6 +65,7 @@ public class PubSubConstants {
   * Default value of sleep interval for polling topic deletion status from ZK.
   */
  public static final int PUBSUB_TOPIC_DELETION_STATUS_POLL_INTERVAL_MS_DEFAULT_VALUE = 2 * Time.MS_PER_SECOND;
+  public static final long UNKNOWN_LATEST_OFFSET = -12345;


is there any significance for this number or any negative number would work?

*nit long should end in L

arjun4084346 · 2024-12-19T21:40:42Z

...e-controller/src/main/java/com/linkedin/venice/controller/VeniceControllerClusterConfig.java

@@ -826,7 +826,7 @@ public VeniceControllerClusterConfig(VeniceProperties props) {
    this.parentControllerMaxErroredTopicNumToKeep = props.getInt(PARENT_CONTROLLER_MAX_ERRORED_TOPIC_NUM_TO_KEEP, 0);

    this.pushJobStatusStoreClusterName = props.getString(PUSH_JOB_STATUS_STORE_CLUSTER_NAME, "");
-    this.participantMessageStoreEnabled = props.getBoolean(PARTICIPANT_MESSAGE_STORE_ENABLED, false);
+    this.participantMessageStoreEnabled = props.getBoolean(PARTICIPANT_MESSAGE_STORE_ENABLED, true);


Why this change?

arjun4084346 · 2024-12-19T21:56:52Z

services/venice-controller/src/main/java/com/linkedin/venice/controller/VeniceHelixAdmin.java

+   * @return {@code true} if real-time topics should be created; {@code false} otherwise
+   */
+  boolean isRealTimeTopicRequired(Store store, Version version) {
+    if (!store.isHybrid() || !version.isHybrid()) {


no problem with this, but can the newly created version's "hybridness" be different from that of its store ?

sushantmane marked this pull request as draft November 22, 2024 19:51

sushantmane commented Nov 22, 2024

View reviewed changes

...ient/src/main/java/com/linkedin/davinci/kafka/consumer/LeaderFollowerStoreIngestionTask.java Show resolved Hide resolved

lusong64 reviewed Dec 3, 2024

View reviewed changes

...enice-common/src/main/java/com/linkedin/venice/controllerapi/RequestTopicForPushRequest.java Outdated Show resolved Hide resolved

lusong64 reviewed Dec 3, 2024

View reviewed changes

...ces/venice-controller/src/main/java/com/linkedin/venice/controller/server/CreateVersion.java Outdated Show resolved Hide resolved

sushantmane force-pushed the li-hybrid-store-partition-count-issue branch 7 times, most recently from c0d5fa3 to c709994 Compare December 12, 2024 01:59

sushantmane marked this pull request as ready for review December 18, 2024 05:42

sushantmane added 2 commits December 17, 2024 21:51

Address review comments and more test

7f27513

sushantmane force-pushed the li-hybrid-store-partition-count-issue branch from 6547aa6 to 7f27513 Compare December 18, 2024 10:53

arjun4084346 reviewed Dec 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[controller] Fix mismatch between hybrid version partition count and real-time partition count #1338

[controller] Fix mismatch between hybrid version partition count and real-time partition count #1338

sushantmane commented Nov 22, 2024 •

edited

Loading

sushantmane commented Nov 22, 2024

lusong64 Dec 3, 2024

sushantmane Dec 18, 2024 •

edited

Loading

arjun4084346 commented Dec 5, 2024

sushantmane commented Dec 6, 2024 •

edited

Loading

arjun4084346 Dec 19, 2024

arjun4084346 Dec 19, 2024

arjun4084346 Dec 19, 2024

arjun4084346 Dec 19, 2024

[controller] Fix mismatch between hybrid version partition count and real-time partition count #1338

Are you sure you want to change the base?

[controller] Fix mismatch between hybrid version partition count and real-time partition count #1338

Conversation

sushantmane commented Nov 22, 2024 • edited Loading

Ensure real-time topic partition count matches hybrid version partition count

How was this PR tested?

Does this PR introduce any user-facing changes?

sushantmane commented Nov 22, 2024

lusong64 Dec 3, 2024

Choose a reason for hiding this comment

sushantmane Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

arjun4084346 commented Dec 5, 2024

sushantmane commented Dec 6, 2024 • edited Loading

arjun4084346 Dec 19, 2024

Choose a reason for hiding this comment

arjun4084346 Dec 19, 2024

Choose a reason for hiding this comment

arjun4084346 Dec 19, 2024

Choose a reason for hiding this comment

arjun4084346 Dec 19, 2024

Choose a reason for hiding this comment

sushantmane commented Nov 22, 2024 •

edited

Loading

sushantmane Dec 18, 2024 •

edited

Loading

sushantmane commented Dec 6, 2024 •

edited

Loading