Install python packages into venvs across workflows.

[ScottTodd]

Many of these workflows are using persistent self-hosted runners, so it looks like they have been reusing the same system-wide Python environment between workflow runs (plus layer of caching on top). This switches to using venvs at ${{ github.workspace }}/.venv that should be ephemeral, giving us more explicit control over which packages are installed.

More work is planned as part of #584 to refactor these workflows further - replacing the package installs code like pip install --no-compile -r requirements.txt -r sharktank/requirements-tests.txt -e sharktank/ with a setup_venv.py script that uses dev/nightly/stable packages (from an appropriate source).

Good reading: https://adamj.eu/tech/2023/11/02/github-actions-faster-python-virtual-environments/

[ScottTodd]

Notes to self:

Need to be deliberate with pip caching. I think I want the downloads to be cached, but not any installed packages.

Reference https://adamj.eu/tech/2023/11/02/github-actions-faster-python-virtual-environments/ and https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#caching-packages.

Both of those approaches hash the requirements files as the cache key, but we have so many separate requirements files and we install our own dev and nightly packages on top, which are not pinned in any requirement files to be cached.

Some of that will hopefully be less consequential, if/when we switch to the setup_venv.py + uv setup style that I'm prototyping on #625.

[ScottTodd]

Maybe try landing tomorrow without any pip/venv caching, then figure out how to add it back carefully. Some extra CI time is worth the environment consistency.