Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Keep progress markers separate from underlying task data #464

Open
starpit opened this issue Oct 25, 2024 · 0 comments
Open

feat: Keep progress markers separate from underlying task data #464

starpit opened this issue Oct 25, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@starpit
Copy link
Member

starpit commented Oct 25, 2024

Currently, we cannot stream the input stream directly from s3 into the stdin of the worker. This is because currently the inbox->processing->outbox flow (needed by the workstealer) is not separate from the data to be processed. We will need to keep them separate, somehow. So that the inbox->processing->outbox flow is done with markers, and the underlying data is placed in a separate folder. This will also avoid potentially expensive copying of large data blobs through those stages.

@starpit starpit added the enhancement New feature or request label Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant