Enable import module before call setup statement (setuptools & setup.py)

[latot]

Hi all! I was checking a weird problem on the GDAL python bindings.... doing it short.

Actually, setuptools only permits to import a module after call setup with setup_requires with the module we want to import,

Now here is the challenge, in GDAL, numpy is an optional feature, you don't need numpy to install gdal, but if you want to use it, GDAL need to check data of numpy, the included folders, and parse that data to the setup statement....

So... to call setup you need import numpy.
To import numpy, you must call setup

Numpy is optional, should not be installed as a requirement

Any ideas how to handle this?

Thx!

[abravalheri]

If you use "build isolation" (which requires a pyproject.toml file with a [build-system] table) you can have a temporary build environment with build-only dependencies... This way numpy can be made available just for the build, but not installed in the normal runtime environment...

Please check out pyproject.toml's spec for [build-system] requires: https://packaging.python.org/en/latest/specifications/declaring-build-dependencies/

[rouault]

what about when the dependency is a extras_require ?
That is we'd ideally want numpy to be installed (and available in code of setup.py) if "pip install gdal[numpy]" , but not in other cases.
If numpy is asked, the list of ext_modules has an extra module compared to without it, and we must know that before setup() has run

[webknjaz]

Extras are just metadata for installers, the build front-end doesn't know or care about them.
If you want conditional builds, you'll probably need a thin build backend wrapper that adds build deps conditionally, based on config_settings.

[abravalheri]

I don't think the installer (e.g. pip) even passes the "currently being requested extras" information to the build frontend or backend (e.g. setuptools or a wrapper), because it is not something that is used to produce a wheel.

In the end of the day, pip will build a wheel first and then install it in a following step.
It is better to consider all these phases as consecutive and separated (I believe that mental model will be a closer match on how the ecosystem works).

The complete flow is approximately represented in the figure below. If you are installing from PyPI, pip will cut corners by trying to start from the pre-built wheel (or if that is not available, from the sdist).
So making a conditional installation is something complicated that requires, as described by webknjaz, using a backend-wrapper, config_settings and a sdist-only distribution practice; which as a whole is something I would not recommend.

flowchart TB
    a("source tree") -->|"build process, phase 1"| sdist -->|"build process, phase 2"| wheel -->|"glorified unzip + dependency resolution + script wrapper generation"| x("installed on `site-packages` dir")

Loading