Support for stable NixOS & older Python releases?

[Ma27]

A note upfront: this is me with my work-hat on. I'm not reaching out to ask you to do all the work,
but to see if

• others have better solutions for the problems I'm describing.
• whether there's interest to have some of the ideas (or alternatives) in poetry2nix itself.
  I'd be considering to step up as co-maintainer.

---

Context: at work I'm operating numerous applications, some of them written in Python. The challenges here are

• sometimes we cannot upgrade (yet). There's quite a bunch of Python 3.8 stuff and I think we even have older Python
  (I'm only talking about Python 3+ here, I don't even know if poetry is usable with Python 2.7).

• Right now we have NixOS and VirtualEnvs on top. This is a well-known source for pain, mostly due to prebuilt
  wheels assuming a traditional FHS structure. But also source dists are non-trivial because building them on-the-fly
  causes store references in the virtualenv where you need to create explicit gcroots to avoid them being
  garbage-collected.

My hope is that we can build some of these applications with poetry/poetry2nix to have a Nix-native build
circumventing all of these issues (short of benefits such as automatic rebuilds if e.g. the glibc/openssl/...
from the nixpkgs used gets a security update).

The biggest challenge I faced while playing around with poetry2nix is that it's directly following
nixos-unstable:

• For good reasons, nixpkgs upstream drops packages that were EOLed, such as Python 3.8. However, this is
  not always an option. I'm wondering if it would make sense for poetry2nix to pick up the
  Python interpreter specified in pyproject.toml and perhaps even provide a few Python packages
  that were dropped upstreams, similar to what nix-phps is
  doing.

  Would you be willing to accept such a change?

• Generally, I'm not a big fan of using unstable nixpkgs for that kind of work, there's - for me -
  a too high risk of breakage between rev upgrades in contrast to a stable NixOS.

  Would it make sense to maintain a "stable" branch of poetry2nix such as what home-manager or
  nixos-mailserver are doing? This should probably include an up-to-date build-systems.json and
  all build-system fixes being backported.

  I'm considering to help out here.

Finally, I managed to mess a build up pretty quickly because of locked dependencies that break build-systems
because they depend on the same package, but on a different version. See
#1616 for further context. The fix I proposed there seems to work so far, but would be helpful to get more feedback there.

[TyberiusPrime]

Having a keen interested in software archeology, I'm all in favor.
Having the overrides mixed in with the actual poetry2nix code isn't grant, I believe,
and it would probably be a good idea to split it off, so that poetry2nix versions compatible with older nixpkgs (looking at you, 23.11) could still work with python packages that were overriden later on.

[adisbladis]

sometimes we cannot upgrade (yet). There's quite a bunch of Python 3.8 stuff and I think we even have older Python
(I'm only talking about Python 3+ here, I don't even know if poetry is usable with Python 2.7).

For dropped Python interpreters there is https://github.com/cachix/nixpkgs-python/.

I'm wondering if it would make sense for poetry2nix to pick up the Python interpreter specified in pyproject.toml

I'm rewriting poetry2nix, currently out of tree, on top of a pyproject.nix.
The UX will be very similar to what pdm2nix looks like.

I think a function for selecting a Python interpreter based on pyproject.toml metadata makes sense on top of pyproject.nix's project lib.

Having the overrides mixed in with the actual poetry2nix code isn't grant, I believe,

I 100% agree. I've wanted to split this into a separate repo for quite some time.
These overrides are in no way specific to anything Poetry, and could be shared among many projects.

However, I also see a huge opportunity here to improve on what we currently have.
The existing overrides require human intervention, but I think we could skip that in a majority of cases.
We could have a bot that crawls pypi, downloads pyproject.toml to extract build systems, and scan wheels for references to ELF's that we can automatically add as buildInputs.

[TyberiusPrime]

However, I also see a huge opportunity here to improve on what we currently have. The existing overrides require human intervention, but I think we could skip that in a majority of cases. We could have a bot that crawls pypi, downloads pyproject.toml to extract build systems, and scan wheels for references to ELF's that we can automatically add as buildInputs.

My brute-force bot (see #1742) tries it 'the other way':
See if it builds, interpret the error messages, and add what they suggest , repeat until (it builds | the error messages stop changing).

That does get you surprisingly far (plus minus the packages that need a 'sweep' of all version to determine cross-over points when their build-systems changed, working on it).

I suppose the next step (after I've finally landed that PR with all the overrides) is to turn it into a standalone tool that you can throw at a poetry-project and it will suggest the necessary overrides.

Crawling pypi isn't a bad idea, but there's some scaling issues to be aware of.
Machnix did that (for hashes and dependencies) and it exceeded the github repo size limits.
Twice. So you can't just 'create a new auto-build-systems.json' every day.

[adisbladis]

Crawling pypi isn't a bad idea, but there's some scaling issues to be aware of.
Machnix did that (for hashes and dependencies) and it exceeded the github repo size limits.
Twice. So you can't just 'create a new auto-build-systems.json' every day.

To be clear: This is not what I have in mind at all. I don't think it's a good approach.
I'm talking about something far more disciplined and targeted than trying to swoop up all of pypi.