Skip to content
This repository has been archived by the owner on Sep 24, 2024. It is now read-only.

Centralize artifact lineage creation on LMBuddy class #92

Merged
merged 9 commits into from
Apr 4, 2024

Conversation

sfriedowitz
Copy link
Contributor

@sfriedowitz sfriedowitz commented Apr 4, 2024

What's Changing

The key change here is that instead of logging artifact lineages as part of the job entrypoints, I am returning the raw W&B artifacts from the jobs, and logging them at the end of the runs in one location. This offers a few benefits:

  1. We reduce a ton of boilerplate code duplication around when/where to initialize the W&B run.
  2. Because we are not logging artifacts within the job internals, we no longer need to inject ArtifactLoader interface into the entrypoints. This simplifies the code a good amount.
  3. It further standardizes the input/output flow from the entrypoint functions, making them easier for developers to understand and write.
  4. It begins to standardize the role of the LMBuddy class. In a follow-up PR, I plan to refactor the organization of the code a bit to add further functionality to this central class. Much of this is motivated by the design of libraries like Ludwig, and the way in which they use their LudwigModel central class.

@sfriedowitz sfriedowitz marked this pull request as ready for review April 4, 2024 15:03
@sfriedowitz
Copy link
Contributor Author

Confirmed lineage generation working as expected on Ray:
Screenshot 2024-04-04 at 9 56 33 AM

@sfriedowitz sfriedowitz merged commit e1d7f5b into main Apr 4, 2024
4 checks passed
@sfriedowitz sfriedowitz deleted the sfriedowitz/centralized-lineage-generation branch April 4, 2024 16:57
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants