Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support new use case: bootstrap of new streaming query with initial state #29

Open
HeartSaVioR opened this issue Apr 23, 2019 · 1 comment

Comments

@HeartSaVioR
Copy link
Owner

HeartSaVioR commented Apr 23, 2019

See below: actual end user asked the use case on Spark user mailing list.
https://lists.apache.org/thread.html/68226abe4be5ec979eecc3e62e8c05b3036aa0876d555411436a35ab@%3Cuser.spark.apache.org%3E

Initial state can be either specific values or calculate from batch data.

The issue is, structured streaming doesn't have a concept of "initial state" - at least in HDFSBackedStateStore. If the version is 0, it doesn't load the state.

So either we should propose a change to Spark to allow initial state for version 0, or create a new checkpoint which has "batch 0" with empty offset/commit but initial state.

We could try latter (less coupled with upstream) and propose a change on upstream if it doesn't work.

@HeartSaVioR
Copy link
Owner Author

I can't think about the approach of creating offsets/commits for empty batch - offset implementations are vary for data sources. Only one I can think it's possible is let state implementation allows version 0 of state as initial state. Unlike other stuff, it would only work if the state implementation supports version 0, as "supporting version" is not defined in the interface.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant