You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we cannot stream the input stream directly from s3 into the stdin of the worker. This is because currently the inbox->processing->outbox flow (needed by the workstealer) is not separate from the data to be processed. We will need to keep them separate, somehow. So that the inbox->processing->outbox flow is done with markers, and the underlying data is placed in a separate folder. This will also avoid potentially expensive copying of large data blobs through those stages.
The text was updated successfully, but these errors were encountered:
Currently, we cannot stream the input stream directly from s3 into the stdin of the worker. This is because currently the inbox->processing->outbox flow (needed by the workstealer) is not separate from the data to be processed. We will need to keep them separate, somehow. So that the inbox->processing->outbox flow is done with markers, and the underlying data is placed in a separate folder. This will also avoid potentially expensive copying of large data blobs through those stages.
The text was updated successfully, but these errors were encountered: