How does setuptools determine the appropriate site-packages directory?

[adamjstewart]

Hi, I'm a developer for the Spack package manager. Spack is similar to Conda in that we have both Python and non-Python packages, and similar to Nix in that we install all packages to a separate prefix allowing for multiple versions of the same software to be installed.

When dealing with Python installations, we've noticed that third-party Python libraries may be installed in one of several directories:

• lib/pythonX.Y/site-packages on most systems
• lib64/pythonX.Y/site-packages on RHEL/CentOS/Fedora
• lib/pythonX/dist-packages on Debian/Ubuntu

We would like to know this directory ahead of time so that we can set PYTHONPATH appropriately and ensure that Python libraries are installed to the same prefix as the system Python expects. Right now, we're using distutils to determine this:

from distutils.sysconfig import get_python_inc, get_python_lib

for plat_specific in [True, False]:
    print(get_python_inc(plat_specific, prefix=""))
    for standard_lib in [True, False]:
        print(get_python_lib(plat_specific, standard_lib, prefix=""))

We then use these values to explicitly tell setuptools where to install the package to:

$ python setup.py install --root=<prefix> --install-purelib=<not-plat-specific-not-standard-lib> --install-platlib=<plat-specific-not-standard-lib> --install-scripts=bin --install-data= --install-headers=<plat-specific>

This seems to work most of the time, but I've noticed that some packages like cython will install to the --install-platlib directory while others like setuptools will install to the --install-purelib directory. Also, distutils is deprecated in Python 3.10 and will be removed in 3.12, so we would like to move away from this approach. Also also, we don't want to have to run Python if possible since we may be cross-compiling. We could simply recursively search for a particular prefix, but we need to know before installation where libraries will get installed.

My questions are as follows:

1. If I run python setup.py install, how does setuptools determine the appropriate site-packages directory?
2. For ancient packages that still use distutils, is this behavior the same? If not, how does distutils handle things?
3. If we want to ensure that installation prefixes follow the same structure as the main Python installation, do we need to specify --install-* like above?

Pinging other Spack devs who would love to know the answers to how best to package Python libraries: @tgamblin @alalazo @skosukhin @permeakra @wspear @robertu94 @dev-zero

Links to Spack issues and PRs for additional context: spack/spack#24095 spack/spack#21446 spack/spack#24076 spack/spack#24526 spack/spack#25998 spack/spack#26546

[jwodder]

FYI, you should be asking about this question of pip, not setuptools. Setuptools' installation capabilities are deprecated and discouraged nowadays in favor of pip.

[CAM-Gerlach]

Thanks, it looks like the latest stable release of build requires 5 run-time dependencies while the latest stable release of installer requires 2 run-time dependencies. The development version of installer has dropped these dependencies, but also dropped support for older versions of Python.

The deps are still only needed on those older versions of Python (note the conditional in the spec). If you're running on a modern, supported version of Python, they will not be installed even in that version of installer.

Unfortunately, Spack still supports installations with Python 2, as many scientific software libraries have been lax on porting. I'm working on deprecating this in spack/spack#28003 but it will be another full release cycle before we can remove this entirely.

Is that so? Based on my experience as a scientific software developer myself, the scientific Python ecosystem led the way in not only porting to Python 3, but also dropping Python 2. Its been a long time since I've seen a significant scientific Python project that was Python 2 only, and almost all the heavy hitters have dropped Python 2 support by now.

Realistically, any project that hasn't ported to the decade+ old Python 3 likely hasn't even considered moving beyond a legacy Setuptools (or even, shudder, distutils based build system), so the intersection of the two sets is likely zero, and you are thus unlikely to need installer for projects that don't support it with no dependencies.

Once Spack only supports Python 3.7+ installations, I'll definitely be eager to try to port our build system to build/installer. Even if build still has run-time dependencies, I can install them from wheels with installer (assuming they are pure-Python) in order to bootstrap build.

Keep in mind that increasing number of projects (including many of the core PyPA ones) rely on other non-Setuptools-based build systems and wheel-only distributrion, particularly with flit being targeted at scientific libraries, scikit-build aiming for heavy-duty PyData and machine learning packages, and Numpy, Scipy and others switching to Meson, and thus you are going to run into an increasing number of projects, including critical ones, that are uninstallable with deprecated legacy build tooling. It is your choice to support end of life, obsolete Python versions, but I just want to make sure you're fully aware of the growing consequences of doing so for you and your users.

[adamjstewart]

Based on my experience as a scientific software developer myself, the scientific Python ecosystem led the way in not only porting to Python 3, but also dropping Python 2. Its been a long time since I've seen a significant scientific Python project that was Python 2 only, and almost all the heavy hitters have dropped Python 2 support by now.

That's certainly true for things like numpy/scipy/matplotlib, but is definitely not the case for specialty research code, much of which doesn't have the funding to be maintained after it was first released. See here for some examples.

considered moving beyond a legacy Setuptools

Is setuptools considered legacy? My impression was the setuptools was still the recommended build backend, although flit/poetry are also supported.

Keep in mind that increasing number of projects (including many of the core PyPA ones) rely on other non-Setuptools-based build systems and wheel-only distributrion, particularly with flit being targeted at scientific libraries, scikit-build aiming for heavy-duty PyData and machine learning packages, and Numpy, Scipy and others switching to Meson, and thus you are going to run into an increasing number of projects, including critical ones, that are uninstallable with deprecated legacy build tooling. It is your choice to support end of life, obsolete Python versions, but I just want to make sure you're fully aware of the growing consequences of doing so for you and your users.

Yes, I'm aware of the consequences. Spack allows you to have multiple versions of the same library, so concretizing and ensuring that versions of the library are compatible is actually really easy. I'm still trying to move Spack to only support non-EOL versions of Python (spack/spack#28003) but there is significant push back.

[abravalheri]

Is setuptools considered legacy? My impression was the setuptools was still the recommended build backend, although flit/poetry are also supported.

Hi @adamjstewart, setuptools is not considered legacy 😄 (or even deprecated). The project is pretty much alive.

What is considered deprecated is calling python setup.py directly and/or not having a pyproject.toml file with PEP 518 information (also there is a setuptools.build_meta:legacy that is only there as "last ressort").

[adamjstewart]

What is considered deprecated is calling python setup.py directly

Yep, we are moving to pip install (and hopefully someday build/installer) in spack/spack#27798. This will support all versions of Python that pip has ever supported and work even for packages with no setup.py or that don't use setuptools.

[CAM-Gerlach]

That's certainly true for things like numpy/scipy/matplotlib, but is definitely not the case for specialty research code, much of which doesn't have the funding to be maintained after it was first released. See here for some examples.

I see; I've always thought it was rather unfortunate that the funding model is the way it is, and have advocated for change on that front, but I also sympathize with those stuck with it. For an free and open source, volunteer project, you have an equal right to spend your time supporting EoL Python versions as you do dropping them, and I'm glad you've taken the needs of your community into account here.

Is setuptools considered legacy? My impression was the setuptools was still the recommended build backend, although flit/poetry are also supported.

Sorry, I can see why my wording was confusing especially since I ended the parenthetical in the wrong place—I mean to say "legacy-Setuptools- (or, shudder, distutils-) based build system", not "legacy Setuptools (or, shudder, distutils based build system)", referring to the legacy (setup.py-based) Setuptools backend (setuptools.build_meta:__legacy__), rather than the modern PEP 517 (setup.cfg/pyproject.toml-based) one, not Setuptools itself being legacy (though long-term in the decade+ timespan, it seems to be the expectation that the community will eventually move on to one or more of the more modern build systems without all of Setuptools' remaining warts, though that's purely speculative and will be up to package authors to decide).

Also, to be clear, Setuptools is no more officially/generally "recommended" than are Flit, Poetry, Pipenv, PDM or any other standards-conforming packaging tool; each has specific use cases they are appropriate for (e.g. Flit-pure Python libraries, Poetry-heavy duty libraries and applications, Pipenv- applications, Scikit-Build- compiled scientific libraries, etc).

there is a setuptools.build_meta:legacy [sic] that is only there as "last ressort" [sic]

Yeah, this is what I was referring to, and directly calling setup.py/not using pyproject.toml more generally as you also mention; it was quite confusing due to how I structure the syntax, sorry.