micromamba 1.5.10 fails to create environments in parallel when cache has been cleaned with -f before

[b-morgenthaler]

Troubleshooting docs

• My problem is not solved in the Troubleshooting docs

Anaconda default channels

• I do NOT use the Anaconda default channels (pkgs/* etc.)

How did you install Mamba?

Micromamba

Search tried in issue tracker

parallel

Latest version of Mamba

• My problem is not solved with the latest version

Tried in Conda?

I didn't try

Describe your issue

Hi,

We are using micromamba 1.5.10 in our Gitlab CI pipeline. The Gitlab runner is an Ubuntu 22.04 LTS VM (shell executor). The pipeline has several stages: one of the first ones is taking care of creating the environments - one environment per job. At the end, the .post stage is taking care of cleaning the micromamba cache (the created environments are removed already). There is no configuration file existing - so all defaults unless given by the command line options.

Findings

• When the jobs run sequentially (not in parallel), all environments are created successfully
• When I use a different conda lock file (from a different environment file), it's a different package that fails with similar error message (presumably the first package to be installed)
• When I remove the -r from the micromamba clean command in the .post stage, it seems to work in parallel.

Side note regarding Latest version of Mamba: our SCM strategy forbids me to upgrade to micromamba 2.x currently, so I actually don't know if this issue also exists with 2.x.

Here's a minimal gist of the pipeline for better understanding the use case:

variables:
  MICROMAMBA_PATH: ${CI_PROJECT_DIR}/micromamba/linux/bin
  MICROMAMBA: ${MICROMAMBA_PATH}/micromamba
  MAMBA_ROOT_PREFIX: /home/gitlab-runner/micromamba

stages:
  - qa

env_1
  variables:
  stage: qa
  before_script:
    - ${MICROMAMBA} create --no-rc -y -c conda-forge --no-pyc --strict-channel-priority --allow-uninstall -f ${CONDA_LOCK_FILE} -n ${CONDA_NAME_1}
  script:
    - echo "do some things inside the environment"
  after_script:
    - ${MICROMAMBA} -n ${CONDA_NAME_1} --no-rc -y env remove || true

env_2
  variables:
  stage: qa
  before_script:
    - ${MICROMAMBA} create --no-rc -y -c conda-forge --no-pyc --strict-channel-priority --allow-uninstall -f ${CONDA_LOCK_FILE} -n ${CONDA_NAME_2}
  script:
    - echo "do some things inside the environment"
  after_script:
    - ${MICROMAMBA} -n ${CONDA_NAME_2} --no-rc -y env remove || true

env_3
  variables:
  stage: qa
  before_script:
    - ${MICROMAMBA} create --no-rc -y -c conda-forge --no-pyc --strict-channel-priority --allow-uninstall -f ${CONDA_LOCK_FILE} -n ${CONDA_NAME_3}
  script:
    - echo "do some things inside the environment"
  after_script:
    - ${MICROMAMBA} -n ${CONDA_NAME_3} --no-rc -y env remove || true

clean_environment:
  stage: .post
  script:
    - ${MICROMAMBA} clean --all -r -y || true
  when: always

mamba info / micromamba info

libmamba version : 1.5.10
     micromamba version : 1.5.10
           curl version : libcurl/8.10.1 OpenSSL/3.3.2 zlib/1.3.1 zstd/1.5.6 libssh2/1.11.0 nghttp2/1.58.0
     libarchive version : libarchive 3.7.4 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.6
       envs directories : /home/gitlab-runner/micromamba/envs
          package cache : /home/gitlab-runner/micromamba/pkgs
                          /home/gitlab-runner/.mamba/pkgs
            environment : None (not found)
           env location : -
      user config files : /home/gitlab-runner/.mambarc
 populated config files :
       virtual packages : __unix=0=0
                          __linux=5.4.0=0
                          __glibc=2.31=0
                          __archspec=1=x86_64_v4
               channels :
       base environment : /home/gitlab-runner/micromamba
               platform : linux-64

Logs

Executing "step_script" stage of the job script
$ ${MICROMAMBA} create --no-rc -y -c conda-forge --no-pyc --strict-channel-priority --allow-uninstall -f ${CONDA_LOCK_FILE} -n ${CONDA_NAME_1}
Empty environment created at prefix: /home/gitlab-runner/micromamba/envs/unique_name_1
Transaction
  Prefix: /home/gitlab-runner/micromamba/envs/unique_name_1
  No specs added or removed.
  Package                                             Version  Build               Channel                                                             Size
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  [a lot of packages to be installed]
  Summary:
  Install: 232 packages
  Total download: 0 B
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Transaction starting
error    libmamba Error opening for reading "/home/gitlab-runner/micromamba/pkgs/[package].conda": No such file or directory
[package].conda tarball has incorrect checksum
error    libmamba File not valid: SHA256 sum doesn't match expectation "/home/gitlab-runner/micromamba/pkgs/[package].conda"
    Expected: 68d3bbfe418dda8805b7af1e933b45a2ec4064dae53a281201418cad79578b45
    Actual: e3b0c44298fc1c149afbf4c8996fb924ae41e4649b934ca495991b7852b855
    
warning  libmamba '/home/gitlab-runner/micromamba/pkgs/[package].conda' validation failed
critical libmamba Found incorrect download: [package name] Aborting

environment.yml

# please note that we use conda lock files to create environments
# this is an example, the environment file (resp. the conda lock file from it)
# doesn't actually matter or change the error
name: python_build_process
channels:
- [URL to our internal conda-forge proxy from Sonatype Nexus Repository OSS 3.68.1-02]
dependencies:
- python=3.12.*
- pdm==2.19.3
# - add any conda-forge listed package here

~/.condarc

no .condarc/.mambarc in the Gitlab runner user's home

[jjerphan]

Thank you for this report.

Can you also reproduce the problem with micromamba 2.0?

[b-morgenthaler]

Unfortunately not. It's a corporate system where I am not allowed to use micromamba 2.x (yet).

FWIW, I can confirm, though, that's it's also happening on Windows 10 Pro 22H2 (i.e. the same pipeline running on a Windows pwsh Gitlab runner):

$ & $MICROMAMBA_WIN --version
1.5.10
$ & $MICROMAMBA_WIN create --no-rc -y -c conda-forge --no-pyc --strict-channel-priority --allow-uninstall -f $CONDA_LOCK_FILE -n $CONDA_NAME_1
Empty environment created at prefix: C:\home\gitlab-runner\micromamba\envs\unique_name_1
Transaction
  Prefix: C:\home\gitlab-runner\micromamba\envs\unique_name_1
  No specs added or removed.
  Package                                                    Version  Build                  Channel                                                             Size
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Install:    [a lot of packages to be installed]   
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Summary:
  Install: 209 packages
  Total download: 0 B
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Transaction starting
libzlib-1.3.1-h2[46]([OnPrem Gitlab repo root URL]/-/jobs/19986#L46)6b09_2.conda extraction failed
error    libmamba Error when extracting package: remove: Process cannot access file because it is locked by another process.: "C:\home\gitlab-runner\micromamba\pkgs\libzlib-1.3.1-h2466b09_2\info\recipe\parent\test_compile_flags.bat"
critical libmamba recursive_directory_iterator::recursive_directory_iterator: The directory name is invalid.: "C:\home\gitlab-runner\micromamba\pkgs\libzlib-1.3.1-h2466b09_2.conda"

[jjerphan]

We are now only planning to resolve bug on a 1.x anymore, but please let us know if you can use 2.0 and reproduce the issue.