Specifying PYENV for the same version as the base python causes package install failure

[mpgreg]

The current astro-runtime includes python 3.9 as the "base" version. Creating a PYENV with 3.9 overwrites the python binaries which causes some package install failures downstream.

This will fail when the package is installed AFTER the PYENV statement completes.
Dockerfile:

FROM quay.io/astronomer/astro-runtime:7.4.2-base
PYENV 3.9 dbt 
RUN pip install wandb==0.14.1

This will succeed if the package installs before the PYENV statement.

FROM quay.io/astronomer/astro-runtime:7.4.2-base
RUN pip install wandb==0.14.1
PYENV 3.9 dbt 

[mpgreg]

Error logs with the install AFTER the PYENV statement:

[+] Building 9.6s (26/27)                                                                                                                                    
 => [internal] load build definition from Dockerfile                                                                                                    0.0s
 => => transferring dockerfile: 462B                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                       0.0s
 => => transferring context: 34B                                                                                                                        0.0s
 => resolve image config for quay.io/astronomer/airflow-extensions:latest                                                                               0.5s
 => CACHED docker-image://quay.io/astronomer/airflow-extensions:latest@sha256:b54b6ae916a7612111f43f4300aee050cd7524f4c9677b617ad367df54c58f4c          0.0s
 => [astro] transform PYENV directives                                                                                                                  0.0s
 => => transferring dockerfile: 462B                                                                                                                    0.0s
 => [internal] load .dockerignore                                                                                                                       0.0s
 => [internal] load build definition from Dockerfile                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/python:3.9-slim                                                                                      1.2s
 => [internal] load metadata for quay.io/astronomer/astro-runtime:7.4.2-base                                                                            0.1s
 => [auth] library/python:pull token for registry-1.docker.io                                                                                           0.0s
 => [internal] load build context                                                                                                                       0.0s
 => => transferring context: 32.73kB                                                                                                                    0.0s
 => CACHED FROM docker.io/library/python:3.9-slim@sha256:5192f07402cbe8b0267eef13085b321d50ab8aaac79d2f0657f96810c3f4555c                               0.0s
 => CACHED [stage-0  1/15] FROM quay.io/astronomer/astro-runtime:7.4.2-base@sha256:d492467dd5994eb57453c2f91e397d33ce4ba0e0f77102871e59f3a8a1ff6069     0.0s
 => [stage-0  2/15] COPY packages.txt .                                                                                                                 0.0s
 => [stage-0  3/15] RUN if [[ -s packages.txt ]]; then     apt-get update && cat packages.txt | tr '\r\n' '\n' | xargs apt-get install -y --no-install  0.1s
 => [stage-0  4/15] COPY --link --from=python:3.9-slim /usr/local/bin/*3.9* /usr/local/bin/                                                             0.0s
 => [stage-0  5/15] COPY --link --from=python:3.9-slim /usr/local/include/python3.9* /usr/local/include/python3.9                                       0.0s
 => [stage-0  6/15] COPY --link --from=python:3.9-slim /usr/local/lib/pkgconfig/*3.9* /usr/local/lib/pkgconfig/                                         0.0s
 => [stage-0  7/15] COPY --link --from=python:3.9-slim /usr/local/lib/*3.9*.so* /usr/local/lib/                                                         0.0s
 => [stage-0  8/15] COPY --link --from=python:3.9-slim /usr/local/lib/python3.9 /usr/local/lib/python3.9                                                0.2s
 => [stage-0  9/15] RUN /sbin/ldconfig /usr/local/lib                                                                                                   0.1s
 => [stage-0 10/15] RUN ln -s /usr/local/include/python3.9 /usr/local/include/python3.9m                                                                0.2s
 => [stage-0 11/15] RUN mkdir -p /home/astro/.venv/dbt                                                                                                  0.3s
 => [stage-0 12/15] RUN /usr/local/bin/python3.9 -m venv /home/astro/.venv/dbt                                                                          2.2s
 => [stage-0 13/15] COPY requirements.txt .                                                                                                             0.0s 
 => ERROR [stage-0 14/15] RUN if grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt; then     echo >&2 "Do not upgrade by specifying 'apache-a  4.3s
------                                                                                                                                                       
 > [stage-0 14/15] RUN if grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt; then     echo >&2 "Do not upgrade by specifying 'apache-airflow' in your requirements.txt, change the base image instead!";  exit 1;   fi;   pip install --no-cache-dir -r requirements.txt:                                        
#26 0.302 + grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt                                                                                       
#26 0.304 + pip install --no-cache-dir -r requirements.txt                                                                                                   
#26 0.956 Looking in indexes: https://pip.astronomer.io/v2/                                                                                                  
#26 0.956 Looking in links: https://pip.astronomer.io/simple/astronomer-fab-security-manager/, https://pip.astronomer.io/simple/astronomer-airflow-version-check/
#26 0.957 Requirement already satisfied: astro-sdk-python[amazon,snowflake]==1.5.3 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 1)) (1.5.3)
#26 1.417 Collecting astronomer-cosmos==0.5.2
#26 1.529   Downloading astronomer_cosmos-0.5.2-py3-none-any.whl (30 kB)
#26 1.953 Collecting wandb==0.14.1
#26 1.969   Downloading wandb-0.14.1-py3-none-any.whl (2.0 MB)
#26 2.058      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 23.0 MB/s eta 0:00:00
#26 2.065 Requirement already satisfied: pandas==1.5.3 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 4)) (1.5.3)
#26 2.065 Requirement already satisfied: numpy==1.24.2 in /usr/local/lib/python3.9/site-packages (from -r requirements.txt (line 5)) (1.24.2)
#26 2.470 Collecting scikit-learn==1.2.2
#26 2.487   Downloading scikit_learn-1.2.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (9.2 MB)
#26 2.678      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.2/9.2 MB 48.6 MB/s eta 0:00:00
#26 3.028 Collecting astro-run-dag
#26 3.044   Downloading astro_run_dag-0.2.5-py3-none-any.whl (8.6 kB)
#26 3.099 Requirement already satisfied: apache-airflow-providers-common-sql in /usr/local/lib/python3.9/site-packages (from astro-sdk-python[amazon,snowflake]==1.5.3->-r requirements.txt (line 1)) (1.3.4)
#26 3.099 Requirement already satisfied: python-frontmatter in /usr/local/lib/python3.9/site-packages (from astro-sdk-python[amazon,snowflake]==1.5.3->-r requirements.txt (line 1)) (1.0.0)
#26 3.099 Requirement already satisfied: smart-open in /usr/local/lib/python3.9/site-packages (from astro-sdk-python[amazon,snowflake]==1.5.3->-r requirements.txt (line 1)) (6.3.0)
#26 3.100 Requirement already satisfied: attrs>=20.0 in /usr/local/lib/python3.9/site-packages (from astro-sdk-python[amazon,snowflake]==1.5.3->-r requirements.txt (line 1)) (22.2.0)
#26 3.100 Requirement already satisfied: SQLAlchemy>=1.3.18 in /usr/local/lib/python3.9/site-packages (from astro-sdk-python[amazon,snowflake]==1.5.3->-r requirements.txt (line 1)) (1.4.47)
#26 3.101 Requirement already satisfied: pyarrow in /usr/local/lib/python3.9/site-packages (from astro-sdk-python[amazon,snowflake]==1.5.3->-r requirements.txt (line 1)) (8.0.0)
#26 3.101 Requirement already satisfied: apache-airflow>=2.0 in /usr/local/lib/python3.9/site-packages (from astro-sdk-python[amazon,snowflake]==1.5.3->-r requirements.txt (line 1)) (2.5.3+astro.1)
#26 3.102 Requirement already satisfied: snowflake-connector-python[pandas]<3.0.0 in /usr/local/lib/python3.9/site-packages (from astro-sdk-python[amazon,snowflake]==1.5.3->-r requirements.txt (line 1)) (2.9.0)
#26 3.103 Requirement already satisfied: snowflake-sqlalchemy>=1.2.0 in /usr/local/lib/python3.9/site-packages (from astro-sdk-python[amazon,snowflake]==1.5.3->-r requirements.txt (line 1)) (1.4.7)
#26 3.103 Requirement already satisfied: apache-airflow-providers-snowflake in /usr/local/lib/python3.9/site-packages (from astro-sdk-python[amazon,snowflake]==1.5.3->-r requirements.txt (line 1)) (4.0.4)
#26 3.104 Requirement already satisfied: apache-airflow-providers-amazon>=5.0.0 in /usr/local/lib/python3.9/site-packages (from astro-sdk-python[amazon,snowflake]==1.5.3->-r requirements.txt (line 1)) (6.2.0)
#26 3.105 Requirement already satisfied: s3fs in /usr/local/lib/python3.9/site-packages (from astro-sdk-python[amazon,snowflake]==1.5.3->-r requirements.txt (line 1)) (2021.7.0)
#26 3.128 Requirement already satisfied: jinja2>=3.0.0 in /usr/local/lib/python3.9/site-packages (from astronomer-cosmos==0.5.2->-r requirements.txt (line 2)) (3.1.2)
#26 3.128 Requirement already satisfied: filelock in /usr/local/lib/python3.9/site-packages (from astronomer-cosmos==0.5.2->-r requirements.txt (line 2)) (3.10.7)
#26 3.147 Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/site-packages (from wandb==0.14.1->-r requirements.txt (line 3)) (4.5.0)
#26 3.148 Requirement already satisfied: PyYAML in /usr/local/lib/python3.9/site-packages (from wandb==0.14.1->-r requirements.txt (line 3)) (6.0)
#26 3.148 Requirement already satisfied: setuptools in /usr/local/lib/python3.9/site-packages (from wandb==0.14.1->-r requirements.txt (line 3)) (66.1.1)
#26 3.148 Requirement already satisfied: setproctitle in /usr/local/lib/python3.9/site-packages (from wandb==0.14.1->-r requirements.txt (line 3)) (1.3.2)
#26 3.149 Requirement already satisfied: psutil>=5.0.0 in /usr/local/lib/python3.9/site-packages (from wandb==0.14.1->-r requirements.txt (line 3)) (5.9.4)
#26 3.467 Collecting pathtools
#26 3.490   Downloading pathtools-0.1.2.tar.gz (11 kB)
#26 3.502   Preparing metadata (setup.py): started
#26 3.743   Preparing metadata (setup.py): finished with status 'error'
#26 3.747   error: subprocess-exited-with-error
#26 3.747   
#26 3.747   × python setup.py egg_info did not run successfully.
#26 3.747   │ exit code: 1
#26 3.747   ╰─> [26 lines of output]
#26 3.747       /tmp/pip-install-8vil1_v6/pathtools_c7ed6a1c699a4cc8bbd9ad12e573dd5a/setup.py:25: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
#26 3.747         import imp
#26 3.747       /usr/local/lib/python3.9/site-packages/setuptools/dist.py:798: SetuptoolsDeprecationWarning: As setuptools moves its configuration towards `pyproject.toml`,
#26 3.747       `setuptools.config.parse_configuration` became deprecated.
#26 3.747       
#26 3.747       For the time being, you can use the `setuptools.config.setupcfg` module
#26 3.747       to access a backward compatible API, but this module is provisional
#26 3.747       and might be removed in the future.
#26 3.747       
#26 3.747         parse_configuration(
#26 3.747       Traceback (most recent call last):
#26 3.747         File "<string>", line 2, in <module>
#26 3.747         File "<pip-setuptools-caller>", line 34, in <module>
#26 3.747         File "/tmp/pip-install-8vil1_v6/pathtools_c7ed6a1c699a4cc8bbd9ad12e573dd5a/setup.py", line 42, in <module>
#26 3.747           setup(name='pathtools',
#26 3.747         File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line 152, in setup
#26 3.747           _install_setup_requires(attrs)
#26 3.747         File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line 145, in _install_setup_requires
#26 3.747           dist.parse_config_files(ignore_option_errors=True)
#26 3.747         File "/usr/local/lib/python3.9/site-packages/setuptools/dist.py", line 798, in parse_config_files
#26 3.747           parse_configuration(
#26 3.747         File "/usr/local/lib/python3.9/site-packages/setuptools/config/__init__.py", line 29, in _wrapper
#26 3.747           return fn(*args, **kwargs)
#26 3.747         File "/usr/local/lib/python3.9/site-packages/setuptools/config/setupcfg.py", line 175, in parse_configuration
#26 3.747           distribution._referenced_files.update(
#26 3.747       AttributeError: 'MinimalDistribution' object has no attribute '_referenced_files'
#26 3.747       [end of output]
#26 3.747   
#26 3.747   note: This error originates from a subprocess, and is likely not a problem with pip.
#26 3.748 error: metadata-generation-failed
#26 3.748 
#26 3.748 × Encountered error while generating package metadata.
#26 3.748 ╰─> See above for output.
#26 3.748 
#26 3.748 note: This is an issue with the package mentioned above, not pip.
#26 3.748 hint: See above for details.
#26 4.220 WARNING: You are using pip version 22.3.1; however, version 23.0.1 is available.
#26 4.220 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
------
executor failed running [/bin/bash -o pipefail -e -u -x -c if grep -Eqx 'apache-airflow\s*[=~>]{1,2}.*' requirements.txt; then     echo >&2 "Do not upgrade by specifying 'apache-airflow' in your requirements.txt, change the base image instead!";  exit 1;   fi;   pip install --no-cache-dir -r requirements.txt]: exit code: 1
Error: command 'docker build -t airflow-wandb-demo_b6c032/airflow:latest failed: failed to execute cmd: exit status 1

[mpgreg]

This appears to be caused by installing python3.9 over the top of the base python version. The specific install error here is likely irrelevant as many other issues might arise from overwriting the python binaries.

addPythonVersion() should check if the requested version is already installed. This may not be possible without build-args so it may be necessary to assume python3.9 as the base and hard code a check.

astro-provider-venv/buildkit/internal/transform/transforms.go

Line 199 in f84f6f6

func (r *Transformer) addPythonVersion(venv *virtualEnv) (*parser.Node, error) {

[cdabella]

Related issue, this also blocks installing multiple virtual environments using the PYENV directive.

[ashb]

@cdabella Installing multiple venvs is fixed in #32.

The original error is a problem with the module installed, for instance this works fine:

FROM quay.io/astronomer/astro-runtime:7.4.2-base
PYENV 3.9 dbt 
RUN pip install build wheel