Convert ci-shark-ai.yml
to pkgci_shark_ai.yml
.
#625
+623
−79
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This builds on #589 to make progress on #584.
On the surface, this adds complexity right now. The benefits are:
This also switches to using Python venvs, tightening configuration control in workflows and ensuring that only the expected packages are installed and used for tests, without leaking installed packages across workflow runs. That does increase the workflow time though.
Metrics using
pip
Logs before: https://github.com/nod-ai/shark-ai/actions/runs/12058325322
Install pip deps
(no venv, reusing already installed packages and getting the wrong IREE versions)Run LLM Integration Tests
Logs after (cold cache): https://github.com/nod-ai/shark-ai/actions/runs/12059301876/attempts/1?pr=625
Build shortfin
(cold cache)Setup venv
(from scratch, no deps reused)Install nightly IREE packages
Run LLM Integration Tests
Logs after (warm cache): https://github.com/nod-ai/shark-ai/actions/runs/12059301876/attempts/2?pr=625
Build shortfin
(warm cache, 46% cache hit rate)Setup venv
(from scratch, no deps reused)Install nightly IREE packages
Run LLM Integration Tests
Metrics using
uv
uv
(https://docs.astral.sh/uv/) can be used as an alternative topip
Logs (cold cache): https://github.com/nod-ai/shark-ai/actions/runs/12147344797?pr=625
Build shortfin
(cold cache)Setup venv
(from scratch, no deps reused)Install nightly IREE packages
Run LLM Integration Tests
(failed)Logs (warm cache): https://github.com/nod-ai/shark-ai/actions/runs/12147452119?pr=625
Build shortfin
(cold cache)Setup venv
(from scratch, no deps reused)Install nightly IREE packages
Run LLM Integration Tests
(failed)