Installing Python3.12 causes import error: No module named 'distutils'

[Ryanf55]

I've come across some concerning behavior when you install python3.12-dev on Ubuntu 22.04. It breaks ament_cmake_python due to the removal of distutils! I was hoping to work with a friend who uses MacOS and we had some python3.12 specific issues, so naturally, I just added it to my system.

ryan@B650-970:~/Dev/ros2_ws/src/ardupilot$ sudo apt install python3.12-dev
...

But now, if you try to colcon build, I am no longer able to build.

ryan@B650-970:~/Dev/ros2_ws/src/ardupilot$ colcon build
...
Traceback (most recent call last):
  File "/home/ryan/Dev/ros2_ws/src/ardupilot/build/ardupilot_msgs/ament_cmake_python/ardupilot_msgs/setup.py", line 1, in <module>
    from setuptools import find_packages
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'
gmake[2]: *** [CMakeFiles/ament_cmake_python_build_ardupilot_msgs_egg.dir/build.make:70: CMakeFiles/ament_cmake_python_build_ardupilot_msgs_egg] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:447: CMakeFiles/ament_cmake_python_build_ardupilot_msgs_egg.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
gmake: *** [Makefile:146: all] Error 2
---
Failed   <<< ardupilot_msgs [1.93s, exited with code 2]

Summary: 0 packages finished [2.51s]
  1 package failed: ardupilot_msgs
  1 package had stderr output: ardupilot_msgs
  2 packages not processed

This is super odd, because my system python3 still refers to the 3.10.12 version that is default with Ubuntu 22, not the optional 3.12 I just added.

ryan@B650-970:~/Dev/ros2_ws/src/ardupilot$ which python3
/usr/bin/python3
ryan@B650-970:~/Dev/ros2_ws/src/ardupilot$ python3 --version
Python 3.10.12

Perhaps there is a way using apt to indicate that you can't have ros-humble-ament-cmake-python/jammy,now 1.3.7-1jammy.20240125.200351 and python3.12-dev at the same time, much like you can't have gazebo classic and gazebo harmonic binaries at the same time.

See the below linked thread for a potential workaround. My hope in filing this issue is perhaps there is something that can be added to provide visibility to the issue if others run into it. Perhaps there can be better protection in the build tooling to prevent users from installing python3.12 or detect the user is using python3.12 at build time and fail more gracefully.

Originally posted by @Ryanf55 in #382 (comment)

[cottsay]

The short story is that this behavior will improve when ros2/ros2#1524 lands.

The long story is a bit more complicated. The default logic of find_package(Python3) on Linux is to return the latest version of Python found on the system. For your system, the "default" interpreter appears to be Python 3.10, but you also have Python 3.12 present. The CMake logic therefore finds the Python 3.12 installation and tries to use that, since it's the latest. There are a few problems with this:

1. This isn't necessarily the same interpreter that's used by ament_python packages. For those, colcon will use the same interpreter that was used when it was built.
2. System package dependencies, such as Python packages installed via apt using rosdep, will probably target the system's default interpreter.

The reason you're seeing a problem with distutils is really just the tip of a mountain of errors given this configuration. It happens that the version of setuptools shipped in Jammy is incompatible with Python 3.12. The way Debian/Ubuntu package "purelib" Python packages, the Python 3.12 interpreter is able to "see" all of the system packages intended for Python 3.10. This wouldn't work with the "platlib" (C extension) Python packages, so the Python 3.12 interpreter may behave unreliably because of this.

The workaround, until we land the changes to handle this scenario, is to pass --cmake-args -DPython3_EXECUTABLE=/usr/bin/python3 to your colcon builds.

This behavior has always been present, but for some reason lots of folks are ending up with newer Python interpreters installed on their systems than the "default" one a lot more often.

[cottsay]

If your intent was to actually use Python 3.12 to build ROS on your system, then there is a lot more work to do. When you use rosdep to install dependencies, Python packages will be installed which will only work with the system's default Python interpreter. Even if you get Python 3.12 from apt, you still won't be able to "see" some of those Python packages.

In short, you'll need to manually verify and install all missing Python dependencies for ROS using pip on Python 3.12 specifically.

[mikepurvis]

Not sure if this is the ticket for it, but a big issue coming with Python 3.12 (in Ubuntu Noble!) is that setuptools now screams a warning when it's directly invoked via a setup.py file rather than built into a wheel and installed that way. I started to address this for catkin in ros/catkin#1196, but it's far from the complete story, and the currently-proposed solution in there does some funky stuff with rebuilds and egg-info directories getting littered into the source dirs.