Preventing long run time when Python packages are required

[FlorianPfaff]

Hi,
this is really a useful and great tool and thanks to the developers! Maybe someone here in the community has a suggestion for me? That would be great.
I am not an expert in GitHub actions (yet?) and have difficulties improving one thing: The run time is pretty bad (>30 m to over 1 h) when installing packages in the pylint env - I do this to eliminate the tons of E0401 warnings (e.g., Unable to import 'numpy' (import-error)).
The command that works but takes up almost all the time is (configured in the .mega-linter.yml config):

PRE_COMMANDS:
  - command: pip install --no-cache-dir -r $GITHUB_WORKSPACE/requirements.txt
    venv: pylint

I took a closer look with pip --verbose and it compiles a lot of stuff to build the wheel. Stuff I tried to get around all of this:

1. Install it in alpine linux with apk add py3-numpy py3-scipy py3-matplotlib. It is not found in the venv
2. Using poetry - does not run properly, "This error originates from the build backend, and is likely not a problem with poetry but with scipy (1.9.3) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "scipy (==1.9.3)"'.)"
3. Create a cache folder in $GITHUB_WORKSPACE before running MegaLinter (which runs in a docker container, I believe) and then trying to use it via pip install --cache-dir $GITHUB_WORKSPACE/cache-directory -r $GITHUB_WORKSPACE/requirements.txt. I get an error that it does not have the proper access rights or so.
4. Create a cache folder in $GITHUB_WORKSPACE and mount it via

        with:
          args: -v ${{ github.workspace }}/cache-directory:/cache-directory

in the MegaLinter Step (uses: oxsecurity/megalinter@v6) in the mega-linter.yml (the github action .yml, not the .mega-linter.yml config).
However, /cache-directory is not found in the container.

Any other advice? What is the proper way of doing this? Just accept 1 h run time? Or disable E0401 altogether? I was also thinking about why it does not simply use wheel files, but even when I upgrade pip and wheel before installing, it does build the wheel itself. Considering this also causes it to fail when using the Python flavor of MegaLinter (since it lacks g++) I somehow think not everyone is doing things as I am.

Any ideas? Thanks!

[echoix]

When python wheel packages are being compiled, it is because there is no compatible wheel for the package on the platform where it has to be installed. Our images are based off Alpine Linux, and Alpine Linux uses the musl C library. However, many python packages don't release wheels for musl, even if it possible and some do. These super long build times are even more obvious in our work to build linux/arm64 containers on a linux/amd64 runner, the build times are painful.

[echoix]

As an alternative, do you know if you can cache the built wheels with the action's cache, and try to install from them when installing from your pre-commands? A wheel file is supposed to be a single compressed file where all the job is done!

[Kurt-von-Laven]

Yeah, caching definitely seems the like the best path forward to me. Can you share your code and the error you are facing wherever you get stuck on your best attempt at caching the dependencies? See https://github.com/actions/cache/blob/main/examples.md#python---pip for guidance.

[FlorianPfaff]

Thanks, both of you. Your comments helped me understand the problem better and motivated me to go on. I got it to work.
I used an action to get alpine with a shell running (on top of ubuntu latest), enabled caching, built the wheel there, put them into a subfolder of the repo, and then run mega-linter with the pre command pip install $GITHUB_WORKSPACE/alpine-wheels/* in the pylint venv.
If you want to see the result: mega-linter.yml for the action in .github/workflows and .mega-linter.yml for the configuration in the root of the repository.

If anyone can use an example that only builds the wheels for alpine, see below. It includes all requirements to build the wheels for numpy, scipy, and some other libraries.
If anyone from OxSecurity can use anything of my code to improve mega-linter by making it easier for people to use caching for Python, you can license my code under the license you want (in this case: credit appreciated but not required if not deemed sufficiently sophisticated).

name: Build wheel on Alpine Linux

on:
  push:

jobs:
  build-wheel:
    runs-on: ubuntu-latest

    steps:
    - name: Check out repository
      uses: actions/checkout@v3

    - name: Cache wheel directory
      id: cache-wheels
      uses: actions/cache@v3
      with:
        path: ${{ github.workspace }}/alpine-wheels
        key: ${{ runner.os }}-alpine-wheels-${{ hashFiles('requirements.txt') }}
        restore-keys: |
          ${{ runner.os }}-alpine-wheels-
    - name: Set up Alpine Linux
      uses: jirutka/setup-alpine@v1
      with:
        packages: >
          build-base
          python3-dev
          py3-pip
          gfortran
          openblas-dev
    - name: Upgrade pip and install wheel
      run: |
        python -m pip install --upgrade pip
        python -m pip install wheel
      shell: alpine.sh {0}

    - name: Remove version settings
      if: steps.cache-wheels.outputs.cache-hit != 'true'
      run: |
        sed 's/==.*//' requirements.txt > requirements_no_version.txt
      shell: alpine.sh {0}

    - name: Build wheels
      if: steps.cache-wheels.outputs.cache-hit != 'true'
      run: |
        mkdir -p alpine-wheels
        for package in $(cat requirements_no_version.txt); do
          if ! pip download --only-binary=:all: --no-deps "$package" -d /tmp > /dev/null 2>&1; then
            echo "Building wheel for $package"
            pip wheel --wheel-dir=./alpine-wheels "$package"
          else
            echo "Compatible wheel for $package found, skipping build"
          fi
        done
      shell: alpine.sh {0}

    - name: List built wheels
      run: ls -l ./alpine-wheels