Cannot use custom extension types in C extensions for free-threading python

[shinyano]

Bug report

Bug description:

Python3.13t seems to be unable to use C extensions which use new extension types?

According to chapter 2.1. The Basics in official guide, I wrote 3 files: custom.c, pyproject.toml and setup.py which were identical to the code provided in this tutorial.

In the same location I executed D:\python3.13\python3.13t.exe -m pip install . The extension got installed successfully but when I executed import custom in python3.13t.exe, Python just exited without any output or errors:

C:\Users\zhang>python3.13t
Python 3.13.0 experimental free-threading build (tags/v3.13.0:60403a5, Oct  7 2024, 09:53:29) [MSC v.1941 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import custom

C:\Users\zhang>

Then I tried pip install with stable version of 3.13, and successfully imported custom module:

C:\Users\zhang>D:\python3.13\python.exe
Python 3.13.0 (tags/v3.13.0:60403a5, Oct  7 2024, 09:38:07) [MSC v.1941 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import custom
>>> ^Z

C:\Users\zhang>

I'm using Windows11 and Python3.13, setuptools==75.2.0. The complete code is as follows:

custom.c:

// custom.c

#define PY_SSIZE_T_CLEAN
#include <Python.h>

typedef struct {
    PyObject_HEAD
    /* Type-specific fields go here. */
} CustomObject;

static PyTypeObject CustomType = {
    .ob_base = PyVarObject_HEAD_INIT(NULL, 0)
    .tp_name = "custom.Custom",
    .tp_doc = PyDoc_STR("Custom objects"),
    .tp_basicsize = sizeof(CustomObject),
    .tp_itemsize = 0,
    .tp_flags = Py_TPFLAGS_DEFAULT,
    .tp_new = PyType_GenericNew,
};

static PyModuleDef custommodule = {
    .m_base = PyModuleDef_HEAD_INIT,
    .m_name = "custom",
    .m_doc = "Example module that creates an extension type.",
    .m_size = -1,
};

PyMODINIT_FUNC
PyInit_custom(void)
{
    PyObject *m;
    if (PyType_Ready(&CustomType) < 0) 
        return NULL;

    m = PyModule_Create(&custommodule);
    if (m == NULL)
        return NULL;

    if (PyModule_AddObjectRef(m, "Custom", (PyObject *) &CustomType) < 0) {
        Py_DECREF(m);
        return NULL;
    }

    return m;
}

pyproject.toml:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "custom"
version = "1"

setup.py:

from setuptools import Extension, setup
setup(ext_modules=[Extension("custom", ["custom.c"])])

CPython versions tested on:

3.13

Operating systems tested on:

Windows

[Zheaoli]

Would you mind to build the CPython with --pydebug flag and run the code and post the result here?

I can test it later.

[shinyano]

Would you mind to build the CPython with --pydebug flag and run the code and post the result here?

I can test it later.

Sure, I'll try this out. Should I use 3.13 source code or newest?

[Zheaoli]

Sure, I'll try this out. Should I use 3.13 source code or newest?

Both would be great, But you can try 3.13 branch first!

[colesbury]

Hi @shinyano, thanks for the bug report. The Windows installer has an issue where not all the required macros are defined for the free threaded build. You can work around it by adjusting your setup.py to define the Py_GIL_DISABLED macro:

from setuptools import Extension, setup
import sysconfig

macros = []
if sysconfig.get_config_var("Py_GIL_DISABLED"):
    macros.append(("Py_GIL_DISABLED", 1))

setup(ext_modules=[Extension("custom", ["custom.c"], define_macros=macros)])

[Zheaoli]

thanks for the bug report. The Windows installer has an issue where not all the required macros are defined for the free threaded build. You can work around it by adjusting your setup.py to define the Py_GIL_DISABLED macro:

Should we update the doc to remind this different or this would be fixed in 3.14?

[colesbury]

I don't think the issue will be fixed with the Windows installer any time soon (if ever).

We should:

• Update https://github.com/pypa/setuptools/ so that it automatically defines the macro on Windows for the free threading build
• Document the limitation in https://docs.python.org/3/howto/free-threading-extensions.html#freethreading-extensions-howto (it's already there)
• Possibly also mention it in https://docs.python.org/3/extending/newtypes_tutorial.html#defining-new-types

[colesbury]

Here is the corresponding issue in setuptools: pypa/setuptools#4662

[Zheaoli]

I don't think the issue will be fixed with the Windows installer any time soon (if ever).

We should:

• Update https://github.com/pypa/setuptools/ so that it automatically defines the macro on Windows for the free threading build
• Document the limitation in https://docs.python.org/3/howto/free-threading-extensions.html#freethreading-extensions-howto (it's already there)
• Possibly also mention it in https://docs.python.org/3/extending/newtypes_tutorial.html#defining-new-types

SGTM

I have noticed that you have already commit to the setuptools. So I will update documentation.

[colesbury]

The PR to distutils turned out to be easy. Let's not bother with changing the newtypes tutorial -- it's already fairly long and I think the distutils/setuptools change will probably happen relatively quickly.

[Zheaoli]

The PR to distutils turned out to be easy. Let's not bother with changing the newtypes tutorial -- it's already fairly long and I think the distutils/setuptools change will probably happen relatively quickly.

SGTM, BTW I think distutils should release a new version ASAP. I will discuss this in distutils repo

[shinyano]

After some test I noticed that I'm doing set Py_GIL_DISABLED=1 wrong. I mistakenly thought that "set Py_GIL_DISABLED=1" meant setting the environment variable Py_GIL_DISABLED to 1, but it's actually a macro definition.

from setuptools import Extension, setup
import sysconfig

macros = []
if sysconfig.get_config_var("Py_GIL_DISABLED"):
    macros.append(("Py_GIL_DISABLED", 1))

setup(ext_modules=[Extension("custom", ["custom.c"], define_macros=macros)])

This is exacly what I needed to do, thank you all for the discussion. And I'm sorry for taking up your time with such a basic error.