Distutils removed in Python3.12

[BenSmithers]

distutils was deprecated and then removed with the release of Python 3.12.

And since homebrew now also distributes that newer version of Python with boost-python - the current build system doesn't quite work in newer versions of MacOS because of the distutils dependency in the configure script here.

I found that using sysconfig and site can workaround the way it's used. Something like this works on my end - but this needs more testing against other Python versions. The lines highlighted in the link above could be replaced with

    PYTHON_INCDIR=`${PYTHON_EXE} -c 'import sysconfig; print(sysconfig.get_config_vars("INCLUDEDIR")[0])'`
    if [ -d "$PYTHON_INCDIR" ]; then
        echo " Found python include dir $PYTHON_INCDIR"
    else
        echo " Unable to locate the python include dir"
        return
    fi

    # This is the directory to which libraries should be installed for python to find them
    PYTHON_MODULEDIR=`${PYTHON_EXE} -c 'import site; print(site.USER_SITE)'`
    if [ "$PYTHON_MODULEDIR" ]; then
        echo " Python module install dir is $PYTHON_MODULEDIR"
    else
        echo " Unable to locate the python module dir"
        return
    fi

    # This is the directory that python claims contains its standard library,
    # which may or may not include the actual libpython
    PYTHON_STDLIBDIR=`${PYTHON_EXE} -c 'import sysconfig; print(sysconfig.get_config_vars("LIBDIR")[0])'`
    # This may contain a suffix which appears after the version like in 'libpython3.6m'
    # See https://www.python.org/dev/peps/pep-3149/#proposal
    PYTHONLIBSUFFIX=`${PYTHON_EXE} -c 'import sysconfig; print(sysconfig.get_config_vars("PY_BUILTIN_MODULE_CFLAGS")' 2>/dev/null`

[cnweaver]

Thanks for spotting this. It turns out this issue can be masked if seuptools is installed for whatever reason, as it smuggles in a replacement distutils.

Rummaging in the python config variables seems to be somewhat brittle, as I don't think any particular expectation is provided for stability version to version of which variables exist or what they are named. I think sysconfig.get_path("include") will be the best way to get the header directory, and that sysconfig.get_path("platstdlib") will be the right way to find the python library itself (with some extra fixup stuff for some versions of python installed by Apple being 'special').

Using site.USER_SITE is definitely the better way to get the module install directory; I've been using it for other projects but had forgotten to propagate the idea back to nuSQuIDS.

The library suffix looks to be the trickiest point; sysconfig.get_config_vars("PY_BUILTIN_MODULE_CFLAGS") produces something wildly diffferent from distutils.sysconfig.build_flags (although in fairness it does more what one would expect; it's highly unclear why build_flags is called that), although checking the value of "ABIFLAGS" might do the trick. I'll have to research this one a bit more.

[cnweaver]

Of course everything in python is terrible, so almost none of that worked as I had hoped, but I think I came up with work-arounds for everything.

@BenSmithers , if you could try out the changes in #43 with Homebrew it would be helpful to make sure it doesn't involve some yet-different, wacky way of installing python that the proposed logic doesn't handle.

One point which is not solved is that while working on this I was reminded that it's actually rather tricky to pick a default for python module installation. There are three relevant types of installation:

1. 'System' install
2. User local install (e.g. by an unprivileged user)
3. Install with a virtual environment

In cases 1 and 3, using site.USER_SITE does the wrong thing and leads to surprises (e.g. a broken install which works for the admin who made it but not other users, or the contrents of a venv leaking out and affecting other environments the user uses). Of course, it is exactly what will typically be wanted in case 2. It's possible to auto-detect case 3 (check whether VIRTUAL_ENV is set), but distinguishing cases 1 and 2 is not obvious. Case 1 is particularly tricky because off of the top of my head it might involve installing to a system location using super user privileges during the install step (which I what I always do for my own work), but it also might involve installing to a location that isn't something standard owned by the OS/root because it is part of building something like a self-contained environment for CVMFS.

@arguelles , do you have any idea how we should resolve this module installation thing? I can (and probably will) add a configure option for it, but even if I do that we would still like to pick a default that works as much of the time as possible. I'd shy away from making site.USER_SITE since it can silently break things, except that it's exactly the users who least understand the configuration details who are most likely to want it.

[BenSmithers]

This worked on my end to fix the distutils problem. My installation attempt followed case 2, though the python build was sent by make python-install to /opt/homebrew/lib/python3.12/site-packages rather than $PREFIX/lib/python3.12/site-packages.

[cnweaver]

That python install location is expected, with the current version of the patch. Is there some reason you would expect $PREFIX/lib/python3.12/site-packages? That will not generally be in the python search path, so while one can of course add it manually, it doesn't seem like what we would want for a default (nor, as far as I can tell, would we have done so previously, unless the python install itself happened to live in $PREFIX).