Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

For multi-step flows, consider having the Jobs as kwargs to allow for flexibility in executor setting #1320

Closed
Andrew-S-Rosen opened this issue Dec 8, 2023 · 0 comments · Fixed by #1322
Closed

For multi-step flows, consider having the Jobs as kwargs to allow for flexibility in executor setting #1320

Andrew-S-Rosen opened this issue Dec 8, 2023 · 0 comments · Fixed by #1322
Assignees
@Andrew-S-Rosen
Labels
enhancement New feature or request

Comments

@Andrew-S-Rosen
Copy link
Member

Andrew-S-Rosen commented Dec 8, 2023 •
edited
Loading

What new feature would you like to see?

See title. Tagging @tomdemeyere --- the proof-of-concept is in #1322. That said, for simple things like the example you raised, I'll mention that calling .__wrapped__ on the dummy @job-decorated function is likely sufficient.
@Andrew-S-Rosen Andrew-S-Rosen added the enhancement New feature or request label Dec 8, 2023
@Andrew-S-Rosen Andrew-S-Rosen self-assigned this Dec 8, 2023
@Andrew-S-Rosen Andrew-S-Rosen mentioned this issue Dec 9, 2023
Merged
Andrew-S-Rosen added a commit that referenced this issue Dec 13, 2023
@Andrew-S-Rosen
Use partial functions in flows to increase user flexibility in settin…
6133849 
…g job attributes (e.g. executor) (#1322)

Closes #1320, although I'm not 100% confident that this is the ideal
approach yet. I'm not comfortable with merging this without further
justification that this is the best mechanism.

Pros:
- There is now full flexibility in setting the attributes of each job in
a workflow, meaning you can customize each executor
- It is clearer which kwargs are being set since they are associated
with the given function
- The user can even define their own custom job for a flow without
needing to create a new flow altogether

Cons:
- It is now clunkier to modify the kwargs of a job in a workflow because
you need to import the underlying job and make a partial function
- The user (likely) has to learn about partial functions from the docs
since it is a new concept to most people
- Cannot be used with Prefect since the kwarg is not JSON serialiazable.
See PrefectHQ/prefect#10135.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Andrew-S-Rosen Andrew-S-Rosen

Labels
enhancement New feature or request
Milestone
No milestone
1 participant
@Andrew-S-Rosen