Use standard extension configuration for CPython 3.12+ by jjhelmus · Pull Request #1180 · astral-sh/python-build-standalone

jjhelmus · 2026-07-17T21:05:30Z

Switch CPython 3.12+ builds to the upstream Setup.stdlib extension-building mechanism while preserving the existing static/shared linkage behavior.

Build pkgconf on macOS so that bundled dependencies can be discovered and configured.
Generate a minimal Setup.local for shared and disabled overrides on CPython 3.12+.
Restore CPython’s post-build extension checks and remove obsolete patches/workarounds.
Preserve SQLite extension loading and macOS symbol hiding.
Explicitly disable the clinic test and xxlimited_3_13 extensions. c.f. Enable _testclinic, _testclinic_limited and xxlimited_3_13 #1185

With this change, extension-modules.yml is still used for the following in CPython 3.12:

Disabling modules
Specifying static vs shared linkage
Validating the extension modules specified in the CPython source archive.
Constructing PYTHON.json

Constructing PYTHON.json for CPython 3.12+ from other sources is a needed follow up.

Makes various patches unnecessary:

closes Upstream patch patch-checksharedmods-disable.patch #869
closes Upstream patch patch-configure-disable-stdlib-mod-3.12.patch #875 (This patch is only applied on Python 3.11)
closes Upstream patch patch-pwd-remove-conditional.patch #876 (this patch is only applied on Python 3.11)

Configuration can now appropriately detect SIMD support in the toolchain and adjusts libhacl flags accordingly.

This does not add shared extension suggested in #227. This could be done with a small change as a follow up.

Configure pkgconf to report the temporary directories of the prefix that is used on macOS during CPython builds.

Build standard-library extensions through CPython's configure and Setup.stdlib mechanism for Python 3.12 and newer. Default to statically linking these modules into libpython/python with a few overrides speficied in a Setup.local file created from extension-modules.yaml. This allows CPython's extension checks to run. Python 3.11 and 3.10 contrinue to use the existing process to configure and build extension modules.

Configuration correctly detect the SIMD support in the toolchain and applies the appropiate flags when building libhacl.

These extensions were not enabled when using a custom build configuration + build. Maintain this policy.

…tion Reverts commit 059aed0

jjhelmus · 2026-08-18T18:30:02Z

With MODULE_BUILDTYPE=static set CPython 3.12+ still builds a handful of extensions modules as shared, specifically:

_ctypes_test
_testimportmultiple
_testmultiphase
_testsinglephase
xxlimited
xxlimited_35
xxlimited_3_13

Using the custom configuration the last three of these are disabled and not built. The first four are build as shared modules except for static builds where they are statically linked. Because of the behavior of Setup.stdlib.in theses three need to be included in Setup.local to restore the static linkage. This should be revisited as disabling these in the static builds may be more prudent.

The existing behavior for 3.10 and 3.11 is to include a shared module, #1222 which is retained.
No shared modules are included in 3.12+ which is replicated here and validated.

…stdlib.in A handful of modules are marked as always building shared which causes these to be built as shared modules in the static builds.

jjhelmus · 2026-08-19T14:45:48Z

This is ready for a review.

There is follow up work to construct PYTHON.json from source artifacts rather than from extension-modules.yml for 3.12+ as the yaml file is no longer the source of truth.

This could be done in this PR, in a stacked PR, or as a fast follow. I'd prefer merging this and following with another PR to avoid a long running branch. The work for building PYTHON.json without the yaml file is straightforward but benefits from refactoring the long derive_setup_local function for clarity.

zanieb · 2026-08-19T14:49:05Z

-# ./configure tries to auto-detect whether it can build 128-bit and 256-bit SIMD helpers for HACL,
-# but on x86-64 that requires v2 and v3 respectively, and on arm64 the performance is bad as noted
-# in the comments, so just don't even try. (We should check if we can make this conditional)
-if [[ -n "${PYTHON_MEETS_MINIMUM_VERSION_3_14}" ]]; then
-    patch -p1 -i "${ROOT}/patch-python-configure-hacl-no-simd.patch"
-fi


Why is this resolved?

Portions of this comment are incorrect, the behavior of configure is safe.

The check that configure performs is to determine if the compiler supports the necessary flags (-msse, -mavx2, etc) to allow the SIMD helpers to be built.

On Linux x86-64 these flags are supported by LLVM so the helpers are built:

cpython-3.14> checking whether C compiler accepts -msse -msse2 -msse3 -msse4.1 -msse4.2... cpython-3.14> yes cpython-3.14> checking for HACL* SIMD128 implementation... cpython-3.14> standard cpython-3.14> checking whether C compiler accepts -mavx2... cpython-3.14> yes cpython-3.14> checking for HACL* SIMD256 implementation... cpython-3.14> standard

A runtime check is performed before dispatching to these function, the availability of these helpers does not change the baseline x86-64 requirement.

On macOS arm64 the check fails and the SIMD helpers are not built:

cpython-3.14> checking whether C compiler accepts -msse -msse2 -msse3 -msse4.1 -msse4.2... no cpython-3.14> checking whether C compiler accepts -mavx2... no

On macOS x86-64 the check succeeds and the universal2 versions of the SIMD helpers are built. The universal2 wrapper prevents the SIMD implementations from being included when building the arm64 portion of the universal2 library but this never comes into play because only the x86-64 portion of the library is built.

cpython-3.14> checking whether C compiler accepts -msse -msse2 -msse3 -msse4.1 -msse4.2... yes cpython-3.14> checking for HACL* SIMD128 implementation... universal2 cpython-3.14> checking whether C compiler accepts -mavx2... yes cpython-3.14> checking for HACL* SIMD256 implementation... universal2

The runtime check is done so these helpers are only used when the host indicates support. When running on native x86-64 hardware this depends on the host CPU. When running on Apple Silicon under Rosetta2, the behavior depends on what features Rosetta advertises. The performance is intended by upstream and can be optimized by the user by using to the build that matches the host architecture.

zanieb · 2026-08-19T14:49:20Z

-# See https://github.com/python/cpython/issues/145810#issuecomment-4068139183
-replace_in_all("-LModules/_hacl", "")


Same as https://github.com/astral-sh/python-build-standalone/pull/1180/changes#r3814055392

This flag is no longer being added to LDFLAGS (see the removal of lines 388-394) so it no longer leaks into sysconfig.

For verification I checked the cpython-3.14-x86_64-unknown-linux-gnu-pgo+lto artifact:

❯ unzip cpython-3.14-x86_64-unknown-linux-gnu-pgo+lto.zip ... ❯ tar xf cpython-3.14.7-x86_64-unknown-linux-gnu-pgo+lto-20260818T1407.tar.zst ❯ cd python/install ❯ rg LModules/_hacl ❯

The pattern that used to be replaced does not appear in sysconfig or elsewhere.

zanieb · 2026-08-19T14:49:54Z

+# CPython uses the macOS SDK's zlib without a zlib.pc file. Keep the -lz in
+# Libs.private, but do not make pkg-config require the missing metadata.
+if [[ "${PYBUILD_PLATFORM}" = macos* ]]; then
+    sed -i '' -e 's/ zlib >= 1.2.3//' tcl.pc.in


Should we be using a patch here instead?

This could be accomplished by a patch but I think that setting TCLTK_CFLAGS and TCLTK_LIBS when building CPython on macOS is cleaner. That approach makes this change unnecessary, matches how CPython's macOS installer builds _tkinter and removes the need for pkgconfig entirely.

This does not remove the need for pkgconfig, it is still used to find other libraries such as libffi. Directly configuring Tcl/Tk on macOS could be done or pkgconfig can be used with this patch. There are tradeoffs with both options but I'm going to go with patching the pkgconfig file here so that no change is needed when upgrading Tcl/Tk versions.

This is now done via an inline patch.

zanieb · 2026-08-19T14:50:05Z

+    sed -i '/^Requires: /a\
+Requires.private: x11' tk.pc.in


Same as https://github.com/astral-sh/python-build-standalone/pull/1180/changes#r3814061876

This is now done via an inline patch.

This reverts commit 8d605bd.

This reverts commit 47a7f34.

jjhelmus · 2026-08-20T14:54:58Z

Reported the missing Libs.private in tk.pc.in upstream, https://core.tcl-lang.org/tk/tktview/3e5d09f0e49755b2c6d9a7a5b94c990968bb9734

The modification to tcl.pc.in is specific to the use case of using the macOS SDK which does not provide pkg-config files. As such this does not warrant a report upstream.

jjhelmus force-pushed the jjh/standard_mod_config_312 branch from 2bc109e to 059aed0 Compare July 21, 2026 15:03

zanieb reviewed Jul 21, 2026

View reviewed changes

Comment thread src/validation.rs Outdated

jjhelmus added platform:darwin Specific to the macOS platform platform:linux Specific to the Linux platform python:3.10 python:3.11 python:3.12 python:3.14 labels Jul 21, 2026

jjhelmus mentioned this pull request Jul 21, 2026

Enable _testclinic, _testclinic_limited and xxlimited_3_13 #1185

Open

jjhelmus added arch:all Select all architectures python:all Select all Python versions build:all python:3.10 python:3.11 python:3.12 python:3.14 and removed python:3.10 python:3.11 python:3.12 python:3.14 build:all python:all Select all Python versions labels Aug 12, 2026

jjhelmus requested a review from zanieb August 13, 2026 22:35

jjhelmus mentioned this pull request Aug 14, 2026

macOS: configure does not support linking against a non-system libffi python/cpython#155813

Open

jjhelmus added 6 commits August 18, 2026 09:20

require zlib on macOS

163441c

add pkgconf and require it on macOS for CPython

defce0f

configure pkgconf to report temp dir

4b5c2e9

Configure pkgconf to report the temporary directories of the prefix that is used on macOS during CPython builds.

build _testclinic, _testclinic_limited and xxlimited_3_13 modules

91f8a5f

remove hacl patch

e089aab

Configuration correctly detect the SIMD support in the toolchain and applies the appropiate flags when building libhacl.

jjhelmus added 8 commits August 18, 2026 09:20

require x11 for tk when static linking

75aaef8

add _testclinic, _testclinic_limited, and xxlimited_3_13 to validation

4c867b2

disable _testclinic[_limited] and xxlimited_3_13 extensions

6f54839

These extensions were not enabled when using a custom build configuration + build. Maintain this policy.

remove _testclinic, _testclinic_limited, and xxlimited_3_13 to valida…

670f889

…tion Reverts commit 059aed0

disable the system libffi check on macOS

25f44c1

use CONFIG_64 for bundled mpdec on macOS for 3.12

3362de8

add ncursesw to panel link probe in 3.12

9311946

validate that static builds contain no shared libs

330490c

include modules in Setup.local when their linkage differs from Setup.…

a77eb2c

…stdlib.in A handful of modules are marked as always building shared which causes these to be built as shared modules in the static builds.

jjhelmus force-pushed the jjh/standard_mod_config_312 branch from 5bcf925 to a77eb2c Compare August 18, 2026 18:32

disable shared modules in static builds check in 3.10, 3.11

901d9f2

jjhelmus requested a review from EliteTK August 18, 2026 20:19

zanieb reviewed Aug 19, 2026

View reviewed changes

jjhelmus added 7 commits August 19, 2026 14:07

directly configure Tcl/Tk on macOS for 3.12+

47a7f34

remove pkgconf package

8d605bd

Revert "remove pkgconf package"

43af7bf

This reverts commit 8d605bd.

Revert "directly configure Tcl/Tk on macOS for 3.12+"

6d546ed

This reverts commit 47a7f34.

use a inline patch in build-tcl.sh

ac6dbd6

use an inline patch in build-tk.sh

49e1f0b

fixup comment on libffi on macOS

2c7047d

jjhelmus requested a review from zanieb August 19, 2026 20:43

zanieb approved these changes Aug 19, 2026

View reviewed changes

jjhelmus merged commit 5b84e8e into main Aug 20, 2026
257 checks passed

jjhelmus deleted the jjh/standard_mod_config_312 branch August 20, 2026 13:40

		# See https://github.com/python/cpython/issues/145810#issuecomment-4068139183
		replace_in_all("-LModules/_hacl", "")

Uh oh!

Uh oh!

Conversation

jjhelmus commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

jjhelmus commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jjhelmus commented Aug 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jjhelmus Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jjhelmus Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jjhelmus commented Aug 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jjhelmus commented Jul 17, 2026 •

edited

Loading

jjhelmus commented Aug 18, 2026 •

edited

Loading

jjhelmus Aug 19, 2026 •

edited

Loading

jjhelmus Aug 19, 2026 •

edited

Loading