tox-conda failing on call

[Zeitsperre]

Hi,

I'm trying to debug a failing CI build that is based on tox + tox-conda. It appears that tox-conda fails at the virtualenv creation step, but the trace back does not say why. I have tried using the conda and the pip versions, as well as combinations of the last two versions, of each tox-based dependency (tox, tox-conda, tox-current-env).

Call:

$ tox -e py39 --curent-env

Traceback:

using tox.ini: /home/runner/work/xscen/xscen/tox.ini (pid 3479)
using tox-3.28.0 from /home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox/__init__.py (pid 3479)
skipping sdist step
py39 uses /home/runner/micromamba-root/envs/xscen/bin/python3.9
py39 start: getenv /home/runner/work/xscen/xscen/.tox/py39
py39 cannot reuse: no previous config /home/runner/work/xscen/xscen/.tox/py39/.tox-config1
py39 create: /home/runner/work/xscen/xscen/.tox/py39
___________________________________ summary ____________________________________
  py39: commands succeeded
  congratulations :)
Traceback (most recent call last):
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox/session/__init__.py", line 203, in cleanup
    yield
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox/session/__init__.py", line 197, in runcommand
    return self.subcommand_test()
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox/session/__init__.py", line 225, in subcommand_test
    run_sequential(self.config, self.venv_dict)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox/session/commands/run/sequential.py", line 9, in run_sequential
    if venv.setupenv():
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox/venv.py", line 649, in setupenv
    status = self.update(action=action)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox/venv.py", line 277, in update
    self.hook.tox_testenv_create(action=action, venv=self)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox_current_env/hooks3.py", line 195, in tox_testenv_create
    rm_venv(venv)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox_current_env/hooks3.py", line 140, in rm_venv
    shutil.rmtree(os.path.dirname(os.path.dirname(link)), ignore_errors=True)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/posixpath.py", line 152, in dirname
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/micromamba-root/envs/xscen/bin/tox", line 8, in <module>
    sys.exit(cmdline())
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox/session/__init__.py", line 44, in cmdline
    main(args)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox/session/__init__.py", line 69, in main
    exit_code = session.runcommand()
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox/session/__init__.py", line 197, in runcommand
    return self.subcommand_test()
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox/session/__init__.py", line 205, in cleanup
    self.hook.tox_cleanup(session=self)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox_current_env/hooks3.py", line 269, in tox_cleanup
    if is_current_env_link(venv):
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox_current_env/hooks3.py", line 124, in is_current_env_link
    python, activate = _python_activate_exists(venv)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/site-packages/tox_current_env/hooks3.py", line 118, in _python_activate_exists
    bindir = os.path.dirname(python)
  File "/home/runner/micromamba-root/envs/xscen/lib/python3.9/posixpath.py", line 152, in dirname
    p = os.fspath(p)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Error: Process completed with exit code 1.

(For more context, see: Ouranosinc/xscen#157)

[AntoineD]

It seems to me that tox-current-env and tox-conda cannot be used together, has it worked before?

[Zeitsperre]

@AntoineD It has. Everything seemed to be working a month ago. Beyond changes to the relevant tox packages (which I've tested against), it isn't clear to me what has changed.

[AntoineD]

What is the purpose of using tox-current-env?

[Zeitsperre]

My use-case is my tox configuration acts as a universal testing configuration for both CI and local development.

I have a base environment that is built with a GitHub CI action, so it's wasteful/long to recreate an environment when I have already initialized the one calling tox with the proper packages. That said, my tox configuration has my testing ensemble (env_vars, pytest calls, etc.), so I would like to use it, rather than have to synchronize my GitHub workflows and tox configurations all the time.

I imagine I could get by with a Makefile for testing, but before resorting to that, I wanted to see if there was anything that could be done to resolve this bug.

[AntoineD]

I'm not sure to get the point of using tox-current-env...

Using a tox env for developing seems to be the purpose of the tox option --develop, have you tried it?

For the long env creation duration on the CI, have you tried to use caching?

[Zeitsperre]

(I realize this is quickly moving beyond the scope of tox-conda and I appreciate your help!)

I've only been using tox-current-env when on CI, so that I don't need to create the environment a second time. provision-with-micromamba uses caching by default, so I've been using it to generate my development environment, but perhaps it makes more sense to use that Action to generate a lightweight environment and ensure that caching is enabled when running tox instead?

Using a tox env for developing seems to be the purpose of the tox option --develop, have you tried it?

Never have. Will refer to the docs!

[AntoineD]

Could you point me to a commit that used to work, and to the part of your CI config where tox-current-env prevents from create the env a second time?

[Zeitsperre]

Last failing CI config with tox-current-env (a few days ago): https://github.com/Ouranosinc/xscen/blob/f32b7139f71f1fc2bbd75df5703bc93da2b771d6/.github/workflows/main.yml

Failing step trace back on CI: https://github.com/Ouranosinc/xscen/actions/runs/4283619254/jobs/7459447397

Working CI config with tox-current-env (~a month ago): https://github.com/Ouranosinc/xscen/blob/404d3e21000ab8a706ca04d2ba598b7eae9eb58a/.github/workflows/main.yml

Working trace back on CI: https://github.com/Ouranosinc/xscen/actions/runs/4147083062/jobs/7173540012

[AntoineD]

Thank you.

In the testing job, I see the option --current-env but I cannot figure out which env it prevents the second creation of since I understand that there are 3 envs that are called once with different python versions. Could you please explain a bit more?

[Zeitsperre]

Sure, thanks again!

So the testing matrix creates three separate instances of conda environments, all initialized with the environment.yml that contains my complex dependencies. tox --current-env is being called on each of those instances, with the only major difference between them being the version of Python used (3.8, 3.9. 3.10).

This same environment.yml is present in the tox.ini (via tox-conda), so current-env is present so that we don't recreate the same three environments that we did at the initialization step.

Roughly, the breakdown looks like this:

• Build is triggered
• Three simultaneous runs are created:
  • Three conda environments are initialized with a GitHub Action that uses caching (faster creation) and an environment.yml with Python3.{8|9|10}
  • tox is called at the testing step
    • In order to prevent another env (based on the same environment.yml) from being created, --curent-env is among the call flags
      • (As of about a month ago, this started causing errors, not entirely sure why)

Does this help explain things a bit?