Thanks for spotting this. It turns out this issue can be masked if seuptools is installed for whatever reason, as it smuggles in a replacement distutils.

Rummaging in the python config variables seems to be somewhat brittle, as I don't think any particular expectation is provided for stability version to version of which variables exist or what they are named. I think sysconfig.get_path("include") will be the best way to get the header directory, and that sysconfig.get_path("platstdlib") will be the right way to find the python library itself (with some extra fixup stuff for some versions of python installed by Apple being 'special').

Using site.USER_SITE is definitely the better way to get the module install directory; I've been using it for other projects but had forgotten to propagate the idea back to nuSQuIDS.

The library suffix looks to be the trickiest point; sysconfig.get_config_vars("PY_BUILTIN_MODULE_CFLAGS") produces something wildly diffferent from distutils.sysconfig.build_flags (although in fairness it does more what one would expect; it's highly unclear why build_flags is called that), although checking the value of "ABIFLAGS" might do the trick. I'll have to research this one a bit more.

Of course everything in python is terrible, so almost none of that worked as I had hoped, but I think I came up with work-arounds for everything.

@BenSmithers , if you could try out the changes in #43 with Homebrew it would be helpful to make sure it doesn't involve some yet-different, wacky way of installing python that the proposed logic doesn't handle.

One point which is not solved is that while working on this I was reminded that it's actually rather tricky to pick a default for python module installation. There are three relevant types of installation:

1. 'System' install
2. User local install (e.g. by an unprivileged user)
3. Install with a virtual environment

In cases 1 and 3, using site.USER_SITE does the wrong thing and leads to surprises (e.g. a broken install which works for the admin who made it but not other users, or the contrents of a venv leaking out and affecting other environments the user uses). Of course, it is exactly what will typically be wanted in case 2. It's possible to auto-detect case 3 (check whether VIRTUAL_ENV is set), but distinguishing cases 1 and 2 is not obvious. Case 1 is particularly tricky because off of the top of my head it might involve installing to a system location using super user privileges during the install step (which I what I always do for my own work), but it also might involve installing to a location that isn't something standard owned by the OS/root because it is part of building something like a self-contained environment for CVMFS.

@arguelles , do you have any idea how we should resolve this module installation thing? I can (and probably will) add a configure option for it, but even if I do that we would still like to pick a default that works as much of the time as possible. I'd shy away from making site.USER_SITE since it can silently break things, except that it's exactly the users who least understand the configuration details who are most likely to want it.

This worked on my end to fix the distutils problem. My installation attempt followed case 2, though the python build was sent by make python-install to /opt/homebrew/lib/python3.12/site-packages rather than $PREFIX/lib/python3.12/site-packages.

That python install location is expected, with the current version of the patch. Is there some reason you would expect $PREFIX/lib/python3.12/site-packages? That will not generally be in the python search path, so while one can of course add it manually, it doesn't seem like what we would want for a default (nor, as far as I can tell, would we have done so previously, unless the python install itself happened to live in $PREFIX).