Latest Pipenv changed behaviour on virtualenv isolation = dependencies are skipped to install

[tumido]

The latest release of Pipenv https://github.com/pypa/pipenv/releases/tag/v2020.5.28 has changed the behaviour of pipenv install pypa/pipenv#3057. As a result, Pipenv no longer installs packages that can be located in its path. This breaks s2i builds, since Pipenv is run in a different context than the actual applications. Let me illustrate:

Trying to install any package, that is already present in the global env, results in Pipenv skipping this package installation into the pipenv's virtual env. That includes all the dependencies of Pipenv itself like six or importlib_metadata or certifi. And these packages are used as a dependency in many bigger libraries, which transitively makes them misbehave and break...

Let's assume we have an app using certifi and chardet. The first named is a Pipenv dependency and the second one is not:

$ echo "import chardet, certifi" > app.py

$ pipenv install certifi chardet

$ grep -A 2 "\[packages\]" Pipfile
[packages]
certifi = "*"
chardet = "*"

$ s2i build -c . registry.access.redhat.com/ubi8/python-38:latest test -e ENABLE_PIPENV=true

$ docker run test
---> Running application from Python script (app.py) ...
Traceback (most recent call last):
  File "app.py", line 1, in <module>
    import chardet, certifi
ModuleNotFoundError: No module named 'certifi'

Now let's see where are the packages installed and what our python path actually is:

$ docker run test find / -type d -name certifi 2>/dev/null | grep -v vendor
/opt/app-root/src/.local/venvs/pipenv/lib/python3.8/site-packages/certifi

$ docker run test find / -type d -name chardet 2>/dev/null | grep -v vendor
/opt/app-root/lib/python3.8/site-packages/chardet

$ docker run test python -c "import sys; print(sys.path)"
['', '/usr/lib64/python38.zip', '/usr/lib64/python3.8', '/usr/lib64/python3.8/lib-dynload', '/opt/app-root/lib64/python3.8/site-packages', '/opt/app-root/lib/python3.8/site-packages']

As you can see, each of the packages is located in different location. One is reachable for the python runtime and one is not.

This issue means that any new build (based on s2i Python with Pipenv and started after the Pipenv update) of an application that shares a single (or more) dependency with Pipenv will fail at build time or at runtime for a really inexplicable reason - because a dependency which is specified in the Pipfile doesn't get installed.

It took me really long to trace this issue all the way back to Pipenv, I'm not sure how to solve this without locking pipenv version though.

[goern]

@fridex

[tumido]

Hotfix: RHEL ubi8 based builder images with the Pipenv version lock patch applied can be sourced from:

• quay.io/tcoufal/python-36
• quay.io/tcoufal/python-38

[frenzymadness]

Thank you! Could you please also open an upstream issue?

A simple reproducer:

# Create two virtual environments
$ python3.8 -m venv pipenv_venv
$ python3.8 -m venv app_venv

# Activate the first one and install pipenv into it
$ source pipenv_venv/bin/activate
(pipenv_venv)$ pip install pipenv
…
(pipenv_venv)$ pip list
Package          Version   
---------------- ----------
appdirs          1.4.4     
certifi          2020.4.5.1
distlib          0.3.0     
filelock         3.0.12    
pip              19.3.1    
pipenv           2020.5.28 
setuptools       41.6.0    
six              1.15.0    
virtualenv       20.0.21   
virtualenv-clone 0.5.4     
(pipenv_venv)$ deactivate 

# Activate the second one and try to install app dependencies
# using pipenv from the first virtual environment
$ source app_venv/bin/activate
(app_venv)$ cat Pipfile 
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
six = "*"
certifi = "*"
chardet = "*"
(app_venv)$ pipenv_venv/bin/pipenv install
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (8cd69a)!
Installing dependencies from Pipfile.lock (8cd69a)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00

# certifi and six are missing
(app_venv)$ pip list
Package    Version
---------- -------
chardet    3.0.4  
pip        19.3.1 
setuptools 41.6.0

[goern]

Hotfix: RHEL ubi8 based builder images with the Pipenv version lock patch applied can be sourced from:

• quay.io/tcoufal/python-36

• quay.io/tcoufal/python-38

@fridex is that something we should port to thoth‘s S2I Images?

[goern]

Hotfix: RHEL ubi8 based builder images with the Pipenv version lock patch applied can be sourced from:

• quay.io/tcoufal/python-36
• quay.io/tcoufal/python-38

BTW, for most of the use cases Thoth's micropipenv should be a very good alternative: https://github.com/thoth-station/micropipenv and it is included in all our S2I images, for example: https://quay.io/repository/thoth-station/s2i-thoth-ubi8-py36?tab=info

And Fedora ;) https://fedora.pkgs.org/rawhide/fedora-x86_64/micropipenv-0.1.6-1.fc33.noarch.rpm.html

@durandom

[fridex]

BTW, for most of the use cases Thoth's micropipenv should be a very good alternative: https://github.com/thoth-station/micropipenv

In case of micropipenv (#368), there could be just one virtual environment for the actual application and micropipenv could be installed from an rpm (during the builder container image build so that it is already included #278).

Anyway, worth to discuss this behavior with Pipenv upstream to see their bits on this - especially long term behavior they expect to support.

[torsava]

I can confirm the reproducer. It works with the old version (2018.11.26) and fails with the new one (2020.5.28).

I have tried to coax the correct behaviour from the new version by using the flag --no-site-packages, but it does not help.

[frenzymadness]

Issue reported upstream: pypa/pipenv#4301

[frenzymadness]

The issue is already fixed upstream: pypa/pipenv#4284

[torsava]

And the fix has been released in a new version of pipenv!

https://github.com/pypa/pipenv/releases/tag/v2020.6.2