Upsert

[jab]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

Given a conda environment name and a conda-lock.yml, I would like to be able to run something like “conda-lock upsert -n envname conda-lock.yml” which would first check if some existing environment with this name already matches the lockfile, if so does nothing, and if not it updates the env to match the lockfile.

Why is this needed?

Currently the best you can do to make sure you have a particular named env that matches a lockfile is delete any existing same-named env and then run a conda-lock install to create a new one to match the lockfile.

Besides wasting time in the case that an existing env already matches the lockfile, this is even more of a problem in the case that a needless conda-lock install fails due to a transient error, leaving the user without any working env till the transient error goes away. (This happened to my users recently due to intermittent spurious checksum mismatch errors.)

What should happen?

(Ideally this would create the new env in a tempdir and only move it on top of any existing env after installation of all packages succeeded, to be a bit more atomic. But just the ability to exit early in the case that an existing env already matches a lockfile would be a big win.)

Additional Context

No response

[maresb]

Thanks @jab for the suggestion! This would be a great feature, but development time is pretty scarce here. I'd recommend either:

• Look into pixi. It's a lot more polished, very actively being developed, and already does clever things regarding updating environments.
• If you want to stick with conda-lock, is this a feature that you'd be interested to contribute?

[jab]

Thanks! This is for work where we’re already standardized on conda and switching to pixi for this use case is not something I can do independently or immediately, but it is certainly something I can look into more.

In the meantime, I’m pressed for time too, but is there a “quick and dirty” approach you could describe that I could maybe try to implement in a couple hours tomorrow? Let me know if there’s some cheap comparison possible of the “conda list…” output (possibly hashed into a single value?) that could be compared to a hash from the lockfile spec, I would be down to give that a shot.

[maresb]

is there a “quick and dirty” approach you could describe that I could maybe try to implement in a couple hours tomorrow?

That's a really good question...

If you don't care about diffing the individual dependencies, that helps a lot. Probably your best bet would be to use conda list --json which produces output looking like

[
    {
        "base_url": "https://conda.anaconda.org/conda-forge",
        "build_number": 1,
        "build_string": "py312hef9b889_1",
        "channel": "conda-forge",
        "dist_name": "zstandard-0.23.0-py312hef9b889_1",
        "name": "zstandard",
        "platform": "linux-64",
        "version": "0.23.0"
    },
    {
        "base_url": "https://conda.anaconda.org/conda-forge",
        "build_number": 0,
        "build_string": "ha6fb4c9_0",
        "channel": "conda-forge",
        "dist_name": "zstd-1.5.6-ha6fb4c9_0",
        "name": "zstd",
        "platform": "linux-64",
        "version": "1.5.6"
    }
]

Note that this will ignore any pip dependencies in your environment. I hope this is okay?

You may need to compute the current platform (e.g. linux-64), and for that you can use this. (Oh, or quicker and dirtier, just parse it from the above output!!! 😂)

You should be able to parse the lockfile, extract the packages for the current platform, parse the URLs for the filenames, strip the .conda or .bz2 extension, and compare with dist_name from the JSON above.

I would recommend avoiding mamba list or micromamba list at this time because they recently released v2, and there are still a bunch of bugs with the JSON output. (I'm currently waiting for these issues to be resolved before releasing conda-lock v3.)

I hope that approach fits with your time constraints, and I hope I didn't forget anything important.