Adding Python venv

[erikbosch]

Needed after implicit update to ubuntu 24.04

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 50.44%. Comparing base (f756202) to head (542a400).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #78   +/-   ##
=======================================
  Coverage   50.44%   50.44%           
=======================================
  Files          31       31           
  Lines       11702    11702           
=======================================
  Hits         5903     5903           
  Misses       5799     5799           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[wba2hi]

Is this really using a pip virtual environment? It looks like the problem is fixed, by installing a separate python installation which prevents the system python installation from being polluted - which might be a workable solution.

That the fix works is obvious. Just want to have a common understanding here :)

[erikbosch]

Guess you are right - as I understand it installs Python and adds it to path. It does not in itself initiate/activate a virtual environment. Setup-python has however some support for caching so in one way I think it may reuse certain parts

[erikbosch]

In some other repos like https://github.com/eclipse-kuksa/kuksa-python-sdk/blob/main/.github/workflows/dash.yaml we actually use a virtual environment. From practical perspectives I generally do no think that the difference will be that big as we typically do not need multiple different Python environments in the same workflow.

[wba2hi]

Guess you are right - as I understand it installs Python and adds it to path. It does not in itself initiate/activate a virtual environment. Setup-python has however some support for caching so in one way I think it may reuse certain parts

Agree. But caching seems to be disabled in our case, because from what I understand it needs to be explicitely enabled by using:

cache: 'pip' // pipenv // 

see: https://github.com/actions/setup-python?tab=readme-ov-file#caching-packages-dependencies

Again, not telling here that we have to use a cache or not use cache (both will solve problems and create others), just pointing out things :)

In some other repos like https://github.com/eclipse-kuksa/kuksa-python-sdk/blob/main/.github/workflows/dash.yaml we actually use a virtual environment. From practical perspectives I generally do no think that the difference will be that big as we typically do not need multiple different Python environments in the same workflow.

I implemented that stuff based on some commands which were provided to me, so take it with a grain of salt :D It's probably still not the best way to handle that stuff. However in that case it was mandatory to have a clean / unpolluted python instance because otherwise we would point out libraries which are required by the system as libraries used by our software.

Personally I'm fine with the approach used here. As said, I just wanted to have a common understanding. The Commit Title somehow suggests that we use a venv and maybe even what we have now is called a venv - I'm no Python expert here. I was just directly thinking about the pip venv thingy and was somehow missing in my head and in this commit where it's activated :)

[SebastianSchildt]

What was the issue with not doing this? Incompatible Python version, or just no pip oob? Generally of course "pinning" some Python version is likely a good idea even if it weren't broken before

[wba2hi]

What was the issue with not doing this? Incompatible Python version, or just no pip oob? Generally of course "pinning" some Python version is likely a good idea even if it weren't broken before

We are using ubuntu-latest and it looks like the underlying ubuntu version was upgraded, which lead to a new python version being applied which warns / fails with an error, when the system python is used for installation of requirements:

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
Error: Process completed with exit code 1.

[wba2hi]

LGTM

[erikbosch]

What was the issue with not doing this? Incompatible Python version, or just no pip oob? Generally of course "pinning" some Python version is likely a good idea even if it weren't broken before

We use ubuntu-latest in most workflows, and github recently changed so that ubuntu-latest points to ubunto-24.04 instead of ubuntu-22.04

And then when doing pip install ... we now get errors like below.

It seems that pip in ubuntu-24.04 is more strict than in 22.04

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

See https://github.com/eclipse-kuksa/kuksa-databroker/actions/runs/11270488560