Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PYBIND11_PLATFORM_ABI_ID Modernization Continued (platforms other than MSVC) #5439

Merged
merged 32 commits into from
Dec 20, 2024

Conversation

rwgk
Copy link
Collaborator

@rwgk rwgk commented Nov 10, 2024

Description

This PR updates PYBIND11_PLATFORM_ABI_ID, which determines cross-extension ABI compatibility, to improve accuracy across all platforms other than MSVC (which was modernized previously in #4953). For context on the importance of PYBIND11_PLATFORM_ABI_ID, please refer to PRs #5296 and #5375.

The accuracy improvements can be seen by comparing the PYBIND11_INTERNALS_ID summaries before and after:

master @ 741d86f (logs_32284154734.zip, logs_32284154873.zip):

cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | cut -d' ' -f1 | sort | uniq -c
     27 __pybind11_internals_vX_clang_libcpp_cxxabi1002__
      5 __pybind11_internals_vX_clang_libstdcpp_cxxabi1002__
     27 __pybind11_internals_vX_gcc_libstdcpp_cxxabi1013__
      3 __pybind11_internals_vX_gcc_libstdcpp_cxxabi1014__
      3 __pybind11_internals_vX_gcc_libstdcpp_cxxabi1016__
      2 __pybind11_internals_vX_gcc_libstdcpp_cxxabi1017__
      5 __pybind11_internals_vX_gcc_libstdcpp_cxxabi1018__
      2 __pybind11_internals_vX_icc_libstdcpp_cxxabi1010__
      6 __pybind11_internals_vX_mingw_libstdcpp_cxxabi1019__
      1 __pybind11_internals_vX_pgi_libstdcpp__

This PR (pr5439_logs_32281771697.zip, pr5439_logs_32281771739.zip):

      6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__
     26 __pybind11_internals_vX_system_libcpp_abi1__
      1 __pybind11_internals_vX_system_libcpp_abi2__
     48 __pybind11_internals_vX_system_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__

Note that the compiler-specific identifiers clang, gcc, icc, pgi no longer appear. ABI compatibility is now determined primarily by libstdcpp (gcc) or libcpp (clang) ABI preprocessor macros.

It is expected that overrides like those in pytorch will not be needed anymore. However, this PR ensures that existing overrides will still function, allowing for gradual community adoption.

Similar to #4953, silent fallbacks for unexpected situations have been replaced with #error ... PLEASE REVISE THIS CODE. directives. For such cases, command-line overrides for the affected macros can serve as temporary workarounds. To eliminate the need for such workarounds, users are encouraged to contribute updates to pybind11/conduit/pybind11_platform_abi_id.h as needed.

For completeness, the counts for MSVC (unchanged):

      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1942__

The work on this PR was informed by:

Suggested changelog entry:

``PYBIND11_PLATFORM_ABI_ID`` (which is used in composing ``PYBIND11_INTERNALS_ID``) was modernized to reflect actual ABI compatibility more accurately.

@rwgk
Copy link
Collaborator Author

rwgk commented Nov 11, 2024

With this PR at commit 14f8425 and logs_30720380481.zip downloaded from here:

$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | cut -d' ' -f1 | sort | uniq -c
     26 __pybind11_internals_vX_clang_libcpp_libcpp1__
     10 __pybind11_internals_vX_clang_libstdcpp_usecxx111__
     40 __pybind11_internals_vX_gcc_libstdcpp_usecxx111__
      2 __pybind11_internals_vX_icc_libstdcpp_usecxx111__
      6 __pybind11_internals_vX_mingw_libstdcpp_usecxx111__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1941__
      1 __pybind11_internals_vX_pgi_libstdcpp__

This doesn't look right, _LIBCPP_ABI_VERSION and _GLIBCXX_USE_CXX11_ABI always appear to be 1.

@robertmaynard I was trying to follow your #4953 (comment) but maybe I'm misunderstanding?

(I want to leave NVHPC for later.)

@wjakob @henryiii @isuruf @cryos for awareness.

@rwgk rwgk marked this pull request as ready for review November 11, 2024 00:43
@robertmaynard
Copy link
Contributor

With this PR at commit 14f8425 and logs_30720380481.zip downloaded from here:

$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | cut -d' ' -f1 | sort | uniq -c
     26 __pybind11_internals_vX_clang_libcpp_libcpp1__
     10 __pybind11_internals_vX_clang_libstdcpp_usecxx111__
     40 __pybind11_internals_vX_gcc_libstdcpp_usecxx111__
      2 __pybind11_internals_vX_icc_libstdcpp_usecxx111__
      6 __pybind11_internals_vX_mingw_libstdcpp_usecxx111__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1941__
      1 __pybind11_internals_vX_pgi_libstdcpp__

This doesn't look right, _LIBCPP_ABI_VERSION and _GLIBCXX_USE_CXX11_ABI always appear to be 1.

@robertmaynard I was trying to follow your #4953 (comment) but maybe I'm misunderstanding?

(I want to leave NVHPC for later.)

@wjakob @henryiii @isuruf @cryos for awareness.

_GLIBCXX_USE_CXX11_ABI is a boolean which represents which ABI for a set of the C++ std containers you are using ( std::string, list? ). It is an independent axis compared to __GXX_ABI_VERSION.

__GXX_ABI_VERSION _GLIBCXX_USE_CXX11_ABI Notes
102 0 Only compatible with 102. Predates _GLIBCXX_USE_CXX11_ABI
1002 0 Compatible with >= 1002 && < 2000 + CXX11 abi of 0
1002 1 Compatible with >= 1002 && < 2000 + CXX11 abi of 1
1013 0 Compatible with >= 1002 && < 2000 + CXX11 abi of 0
1013 1 Compatible with >= 1002 && < 2000 + CXX11 abi of 1

This presumes that GCC will rev the major value of __GXX_ABI_VERSION when it next breaks ABI compat

@isuruf
Copy link
Contributor

isuruf commented Nov 12, 2024

One more thing to fix would be to remove the compiler id _gcc or _clang as they are compatible with each other. I think the current checks are supposed to protect against the C standard library, but is too broad.

@wjakob
Copy link
Member

wjakob commented Nov 15, 2024

I am surprised by __pybind11_internals_vX_msvc_mt_mscver1941__ -- how does the current code manage to generate the number 1941? (I thought 19 was hardcoded).

@rwgk
Copy link
Collaborator Author

rwgk commented Nov 15, 2024

I am surprised by __pybind11_internals_vX_msvc_mt_mscver1941__ -- how does the current code manage to generate the number 1941? (I thought 19 was hardcoded).

It's generated in this line:

# define PYBIND11_BUILD_ABI "_mt_mscver" PYBIND11_PLATFORM_ABI_ID_TOSTRING(_MSC_VER)

@wjakob
Copy link
Member

wjakob commented Nov 17, 2024

Ah, interesting. So for /MT, the whole _MSCVER should be included? @isuruf, can you confirm? I am not doing this in nanobind currently and will have to make a change if so.

@isuruf
Copy link
Contributor

isuruf commented Nov 18, 2024

Yes. /MT means a static linkage of the C runtime, and it's better to have the full version.

rwgk and others added 3 commits November 18, 2024 09:06
Further constrain to GXX_ABI 1002 or greater and less than 2000,
hopefully future proof by summarizing that as `1` along with CXX11 on or
off.
@cryos
Copy link
Contributor

cryos commented Nov 20, 2024

I just pushed a new commit that produces something like

__pybind11_internals_v5_gcc_libstdcpp_gxx_abi_1_usecxx11_1__

It also ensures __GXX_ABI_VERSION is less than 2000, so labeling gxx_abi_1, maybe gxx_abi_1xxx would be better. The thought process was to include that as well should it be bumped in the future.

@rwgk
Copy link
Collaborator Author

rwgk commented Nov 20, 2024

Here are the results extracted from the latest CI logs:

(Click on any job, then the gear that appears in the top-right corner of the window with the log for that job, then "Download log archive".)

$ mkdir junk && cd junk
$ unzip ../logs_31139436091.zip
$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | cut -d' ' -f1 | sort | uniq -c
     26 __pybind11_internals_vX_clang_libcpp_libcpp1__
     10 __pybind11_internals_vX_clang_libstdcpp_gxx_abi_1_usecxx11_1__
     40 __pybind11_internals_vX_gcc_libstdcpp_gxx_abi_1_usecxx11_1__
      2 __pybind11_internals_vX_icc_libstdcpp_gxx_abi_1_usecxx11_1__
      6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1_usecxx11_1__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1942__
      1 __pybind11_internals_vX_pgi_libstdcpp__

On chat @cryos and I decided to go with gxx_abi_1xxx_ instead of gxx_abi_1_, for clarity.

Looking at these:

__pybind11_internals_vX_clang_libstdcpp_gxx_abi_1_usecxx11_1__
__pybind11_internals_vX_gcc_libstdcpp_gxx_abi_1_usecxx11_1__

I think we want to omit clang and gcc here, based on my understanding of suggestions by @robertmaynard and @isuruf.

But what about
__pybind11_internals_vX_icc_libstdcpp_gxx_abi_1_usecxx11_1__
?

Does anyone know if icc-compiled code is ABI-compatible with clang and gcc?

And what should we do about this one? __pybind11_internals_vX_clang_libcpp_libcpp1__

Could/should we just omit these lines

#    elif defined(_LIBCPP_ABI_VERSION) // https://libcxx.llvm.org/DesignDocs/ABIVersioning.html
#        define PYBIND11_BUILD_ABI "_libcpp" PYBIND11_PLATFORM_ABI_ID_TOSTRING(_LIBCPP_ABI_VERSION)

and rely on the new __GXX_ABI_VERSION, _GLIBCXX_USE_CXX11_ABI logic instead?

After discussions with Ralf Grosse-Kunstleve we think these would make
good identifiers that are concise and clear.
Within the `__GXX_ABI_VERSION` block this should always be defined,
guard against unexpected defines and make the error obvious.
@rwgk
Copy link
Collaborator Author

rwgk commented Nov 21, 2024

Here are the results extracted from the latest CI logs:

     26 __pybind11_internals_vX_clang_libcpp_abi1__
     10 __pybind11_internals_vX_clang_libstdcpp_gxx_abi_1xxx_usecxx11_1__
     42 __pybind11_internals_vX_gcc_libstdcpp_gxx_abi_1xxx_usecxx11_1__
      2 __pybind11_internals_vX_icc_libstdcpp_gxx_abi_1xxx_usecxx11_1__
      6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1xxx_usecxx11_1__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1942__
      1 __pybind11_internals_vX_pgi_libstdcpp__

@rwgk
Copy link
Collaborator Author

rwgk commented Nov 24, 2024

The current changes affect icc, therefore it'll be best if we settle the PYBIND11_PLATFORM_ABI_ID also for that compiler in this PR. According to

https://www.intel.com/content/www/us/en/docs/dpcpp-cpp-compiler/developer-guide-reference/2023-2/gcc-compatibility-and-interoperability.html

"C language object files created with the compiler are binary compatible with the GCC and C/C++ language library."

AFAICT, the compiler also links against the systemlibstdcpp, therefore I figure it must be binary compatible for C++ as well.

Upshot: I think we should also omit icc in the PYBIND11_PLATFORM_ABI_ID.

@rwgk
Copy link
Collaborator Author

rwgk commented Nov 24, 2024

I just pushed up the experimental commit fe2dbcb which changes PYBIND11_COMPILER_TYPE.

Rationale:

  • Test for special cases first, system compiler last.

  • Replace "unknown" fall-through with #error, similar to the elimination of fall-throughs for PYBIND11_BUILD_ABI. (Possibly we may want to enable fall-throughs for releases, but always #error in the pybind11 CI jobs.)

  • Assume __MINGW32__ and __CYGWIN__ are not binary compatible with MSVC.

  • Use _system for gcc, clang, icc.

  • Leave __PGI untouched for now. (Possibly after we're sure about gcc, clang, icc, we can also settle the case for PGI in this PR.)

I'll post a new PYBIND11_PLATFORM_ABI_ID overview when all jobs are done and green.

@rwgk
Copy link
Collaborator Author

rwgk commented Nov 25, 2024

Latest PYBIND11_INTERNALS_ID overview (this PR at commit fe2dbcb):

logs_31272968106.zip — https://github.com/pybind/pybind11/actions/runs/11999580827/job/33447666543?pr=5439

$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | c
ut -d' ' -f1 | sort | uniq -c
      6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1942__
      1 __pybind11_internals_vX_pgi_libstdcpp__
     26 __pybind11_internals_vX_system_libcpp_abi1__
     52 __pybind11_internals_vX_system_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__

@rwgk
Copy link
Collaborator Author

rwgk commented Nov 25, 2024

@robertmaynard do you have a moment to look at this PR? (The changes are small and all in just one file.)

  • Is the PYBIND11_COMPILER_TYPE and PYBIND11_BUILD_ABI logic good as-is for gcc and clang (Linux, macOS)?

  • Do we have to worry about _DEBUG under Linux, macOS?

  • Should we leave NVHPC (PGI) for later, or try to handle that in this PR as well?

@robertmaynard
Copy link
Contributor

@robertmaynard do you have a moment to look at this PR? (The changes are small and all in just one file.)

  • Is the PYBIND11_COMPILER_TYPE and PYBIND11_BUILD_ABI logic good as-is for gcc and clang (Linux, macOS)?

Looks good to me

  • Do we have to worry about _DEBUG under Linux, macOS?

_DEBUG is a MSVCism. Linux and macOS have NDEBUG but that doesn't change ABI.

  • Should we leave NVHPC (PGI) for later, or try to handle that in this PR as well?

NVHPC should be ABI compatible with gcc. I am fine either way in how you want to manage that integration from a PR perspective.

@rwgk
Copy link
Collaborator Author

rwgk commented Nov 25, 2024

Thanks a lot @robertmaynard for the review!

NVHPC should be ABI compatible with gcc.

Based on your comment, I added commit 9fc9515:

Add NVHPC (__PGI) to the list of compilers (assumed to be) compatible with system compiler.

@rwgk
Copy link
Collaborator Author

rwgk commented Dec 1, 2024

What I learned from the work on #5439 (comment) got me to add

I don't expect any extra fallout from those two changes. The PYBIND11_STDLIB change consolidates the logic related to libc++ and libstdc++. The logic is now easier to understand and certain to be self-consistent.

@rwgk
Copy link
Collaborator Author

rwgk commented Dec 1, 2024

Just to double-check, the latest PYBIND11_INTERNALS_ID overview (this PR at commit b47be2d) is still identical to that posted under #5439 (comment):

logs_31527439795.zip — https://github.com/pybind/pybind11/actions/runs/12108518322/job/33756607834?pr=5439

$ cat *.txt | grep 'C.. Info: .* __pybind11_internals_v' | sed 's/.* __pybind11_internals_v[0-9]*/__pybind11_internals_vX/' | c
ut -d' ' -f1 | sort | uniq -c
      6 __pybind11_internals_vX_mingw_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__
      3 __pybind11_internals_vX_msvc_md_mscver19__
     34 __pybind11_internals_vX_msvc_md_mscver19_debug__
      3 __pybind11_internals_vX_msvc_mt_mscver1942__
     26 __pybind11_internals_vX_system_libcpp_abi1__
     53 __pybind11_internals_vX_system_libstdcpp_gxx_abi_1xxx_use_cxx11_abi_1__

This makes it possible to achieve these two goals:

* Avoid the leading underscore in `PYBIND11_PLATFORM_ABI_ID` (see pybind#5439 (comment))

* Maintain backward compatibility for use cases as reported under pybind#5439 (comment)

`PYBIND11_INTERNALS_KIND` is removed in this commit to ensure that `PYBIND11_COMPILER_TYPE` is the first element of the `PYBIND11_PLATFORM_ABI_ID`, so that `PYBIND11_COMPILER_TYPE_LEADING_UNDERSCORE` can meaningfully be used as a prefix for `PYBIND11_PLATFORM_ABI_ID` in pybind11/detail/internals.h.
@rwgk
Copy link
Collaborator Author

rwgk commented Dec 2, 2024

This idea randomly crossed my mind when I woke up this morning: e071edc

With that, I think it's best to go ahead and remove PYBIND11_INTERNALS_KIND already in this PR.

(Waiting for CI to finish.)

@rwgk
Copy link
Collaborator Author

rwgk commented Dec 2, 2024

Double-checked again, this PR @ commit e071edc:

@rwgk
Copy link
Collaborator Author

rwgk commented Dec 3, 2024

Hi @isuruf, is there a chance that you could take a quick look, especially with a view on how this will work for Conda? I believe this PR is a massive simplification for the purposes of Conda: in the pybind11 CI, all builds using libstdc++ are mutually compatible, and all builds using libc++ are mutually compatible. See #5439 (comment).

If it looks good to you I'll rewrite the PR description.

Comment on lines 26 to 27
# elif defined(__INTEL_COMPILER) || defined(__clang__) || defined(__GNUC__)
# define PYBIND11_COMPILER_TYPE "system" // Assumed compatible with system compiler.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still feels weird.

How about?

Suggested change
# elif defined(__INTEL_COMPILER) || defined(__clang__) || defined(__GNUC__)
# define PYBIND11_COMPILER_TYPE "system" // Assumed compatible with system compiler.
# elif defined(__GLIBC__) && (defined(__INTEL_COMPILER) || defined(__clang__) || defined(__GNUC__))
# define PYBIND11_COMPILER_TYPE "glibc" // Assumed compatible with system compiler.
# elif defined(__APPLE__) && (defined(__INTEL_COMPILER) || defined(__clang__) || defined(__GNUC__))
# define PYBIND11_COMPILER_TYPE "macos" // Assumed compatible with system compiler.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like this. Trying it out in the CI right now: ca9e699

The code is exactly as suggested, but I revised the comments.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. On second thought, this would break musl, android builds etc. Let's keep the earlier one.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, gee: sorry I missed your last comment before.

I don't know about musl and android builds.

But I just revert ... simpler is better here, I think.

rwgk added 11 commits December 3, 2024 10:47
… more general.

The main motivation is to resolve these "Manylinux on 🐍 3.13t • GIL" and "Pyodide wheel" failures:

```
/__w/pybind11/pybind11/include/pybind11/conduit/pybind11_platform_abi_id.h:35:10: error: #error "Unknown PYBIND11_COMPILER_TYPE: PLEASE REVISE THIS CODE."
   35 | #        error "Unknown PYBIND11_COMPILER_TYPE: PLEASE REVISE THIS CODE."
      |          ^~~~~
```

(All other CI jobs succeeded.)

Further thought:

Essentially, under Linux and macOS the `PYBIND11_COMPILER_TYPE` is only for informational purposes.
ABI compatibility is determined by the libstdc++ or libc++ ABI version.
# define PYBIND11_COMPILER_TYPE "gcc_cygwin"
# elif defined(_MSC_VER)
# define PYBIND11_COMPILER_TYPE "msvc"
# elif defined(__INTEL_COMPILER) || defined(__clang__) || defined(__GNUC__)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the interest of simplification, consider removing defined(__INTEL_COMPILER)

The Intel classic compilers, icc for C and icpc for C++, define __GNUC__ as well as __INTEL_COMPILER.
The last version was 2021.10 and it stopped shipping in 2023.

The "Intel(R) oneAPI DPC++/C++ Compilers", icx for C and icpx for C++, do not define __INTEL_COMPILER.
They do define all of these: __GNUC__, __clang__, __INTEL_CLANG_COMPILER, and __INTEL_LLVM_COMPILER.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: 738b7ec

Thanks @hpkfft for the suggestion!

@wjakob
Copy link
Member

wjakob commented Dec 20, 2024

This PR looks great to me now, are there any other planned changes?

@rwgk
Copy link
Collaborator Author

rwgk commented Dec 20, 2024

This PR looks great to me now, are there any other planned changes?

Only some work on the PR description. Sadly Flu + RSV have wiped out a lot of my time. I'll try to do it tonight.

@rwgk rwgk changed the title PYBIND11_PLATFORM_ABI_ID Modernization Continued (WIP) PYBIND11_PLATFORM_ABI_ID Modernization Continued (platforms other than MSVC) Dec 20, 2024
@rwgk
Copy link
Collaborator Author

rwgk commented Dec 20, 2024

I updated the PR description.

Thanks so much Everybody for all the input and feedback!

@rwgk rwgk merged commit 3e41948 into pybind:master Dec 20, 2024
76 checks passed
@github-actions github-actions bot added the needs changelog Possibly needs a changelog entry label Dec 20, 2024
@rwgk rwgk deleted the pybind11_platform_abi_id_cont branch December 20, 2024 09:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs changelog Possibly needs a changelog entry
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants