Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-plan pipelines when hints change #60

Merged
merged 1 commit into from
Feb 7, 2024
Merged

Re-plan pipelines when hints change #60

merged 1 commit into from
Feb 7, 2024

Conversation

ryannedolan
Copy link
Collaborator

Summary

Closes #53.

Details

The subscription operator now sets .status.hints after planning a pipeline, and re-plans the entire pipeline whenever .status.hints diverges from .spec.hints. This means that users can modify a Subscription's hints, and the changes will take effect. For example, a users can repartition an output Kafka topic by adjusting .spec.hints.kafka.numPartitions.

Testing

I edited an existing Subscription with kubectl edit subs products and added .spec.hints.kafka.numPartitions = 16. The operator reacts accordingly:

--->%---
Planning a new pipeline for Subscription/products with SQL `SELECT "quantity", "product_id" AS KEY FROM INVENTORY."products_on_hand"`...
--->%---
Found existing topic products
--->%---
Desired partitions 16 > actual partitions 1. Creating additional partitions.
--->%---

@ryannedolan ryannedolan merged commit 1755ea7 into main Feb 7, 2024
1 check passed
@ryannedolan ryannedolan deleted the update-hints branch February 7, 2024 04:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sub operator ignores changes to hints
2 participants