diff --git a/.gitignore b/.gitignore index 6c3ef9113..c89ecb7d6 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,3 @@ *.pyc build_artifacts - -.vscode/ diff --git a/abs.yaml b/abs.yaml index a467ff00c..ebffbce4d 100644 --- a/abs.yaml +++ b/abs.yaml @@ -5,9 +5,10 @@ build_parameters: - "--error-overlinking" - "--error-overdepending" -# Upload testing artifacts: ad-testing/label/py315 +# Upload testing artifacts: ad-testing/label/py315-debug (PKG-2624 spike). +# Release 3.15 stays on PR #256 → ad-testing/label/py315. upload_channels: - - ad-testing/label/py315 + - ad-testing/label/py315-debug build_env_vars: ANACONDA_ROCKET_ENABLE_PY315: yes diff --git a/anaconda.yaml b/anaconda.yaml index c98639f2f..a189601f7 100644 --- a/anaconda.yaml +++ b/anaconda.yaml @@ -7,8 +7,10 @@ upstream_sources: from_pattern: ^v(\d+\.\d+\.\d+)$ to_pattern: \1 packages: - - libpython-static - python + - libpython + - libpython-static + - cpython - python-freethreading - python-gil - python-jit diff --git a/recipe/DEBUG.md b/recipe/DEBUG.md new file mode 100644 index 000000000..0a8ac0693 --- /dev/null +++ b/recipe/DEBUG.md @@ -0,0 +1,111 @@ +# Py_DEBUG spike (PKG-2624) + +Time-boxed technical spike stacked on python-feedstock#256. **Not pkgs/main.** Product ticket PKG-2624 is closed (no market). + +## What this is (and is not) + +| Build | What you get | What you do not get | +|-------|----------------|---------------------| +| **Unix** (`--with-pydebug`) | `Py_DEBUG`, `'d' in sys.abiflags`, `sys.gettotalrefcount()`, `libpython3.15d.so` / `libpython3.15td.so` | DWARF-complete unstripped objects (conda-build may strip `.symtab` / `.debug_info`) | +| **Windows** (`PCbuild -d`) | `python315_d.lib` / `python3_d.lib` / `_tkinter_d.lib`, `gettotalrefcount`, debug CRT | Not the same as Unix `abiflags` (`sys.abiflags` is Unix-only) | +| **`python -X dev`** | Runtime development checks on a **release** interpreter | `d` ABI, refcount APIs, debug libpython | + +Do not confuse Py_DEBUG / PCbuild `-d` with `-X dev`. This dest ships debug interpreters on unix **and** win-64. + +## Gate: keep debug off pkgs/main + +PBP `upload_channels` is **graph-wide**. Do **not** zip `channel_targets` with `python_debug` (F94). The dummy `channel_targets = ('abc','def')` in `meta.yaml` exists so the linter expands and `'conda-forge' in channel_targets` is false — leave it. + +| File | Debug spike (this PR) | Before pkgs/main / #256-style py315 | +|------|------------------------|--------------------------------------| +| `recipe/conda_build_config.yaml` | `build_type: debug` only; `freethreading: yes/no`; unix+win | Restore `release` (and drop `debug` unless dest is still a testing label) | +| `abs.yaml` | `upload_channels: [ad-testing/label/py315-debug]` | `#256` keeps `ad-testing/label/py315`. Never point debug variants at `main` | + +Release 3.15 stays on **#256** → `ad-testing/label/py315`. This PR is a **second graph**. + +## Pinning a debug interpreter + +Build string contains `_debug_` plus the ABI tag: + +```text +python 3.15.* *_debug_cp315 # GIL debug +python 3.15.* *_debug_cp315t # free-threading debug +``` + +`run_exports` (when `build_type == debug`) already pins: + +```text +python {{ ver2 }}.* *_debug_{{ abi_tag }} +``` + +Downstream recipes that must link the debug ABI should depend on that pin, **not** on a generic `python 3.15.*`. + +Debug Python **cannot load release extensions** (and vice versa). `abiflags` `d` / `td` vs release `''` / `t` are different ABIs. Mixing them is a loader/ABI error, not a solver hint. + +## Matrix + +- `freethreading: yes/no` × `build_type: debug` +- **No `release` on this graph** +- GIL debug lib: `libpython3.15d.so` (`abiflags` contains `d`) +- FT debug lib: `libpython3.15td.so` (`abiflags` contains `td`; assert `'d' in sys.abiflags` covers both) +- Windows: PCbuild `-d`. Import libs in `libs\`: `python315_d.lib` / `python315t_d.lib`, `python3_d.lib` / `python3t_d.lib`, `_tkinter_d.lib`. Do not add a win **release** variant onto `py315-debug`. + +## Local conda-build + +```bash +export ANACONDA_ROCKET_ENABLE_PY315=yes +cd /home/skupr/src/aggregate/python-feedstock +conda-build recipe --suppress-variables --error-overlinking --error-overdepending \ + -c defaults --override-channels +``` + +CBC is already debug-only on this branch, so omit `--variants`. Prefix: + +```text +conda-bld/linux-64/python-3.15.0rc2-*_debug_cp315*.conda +``` + +On a tree that still has `release` (e.g. #256 HEAD) force one variant: + +```bash +conda-build recipe --suppress-variables --error-overlinking --error-overdepending \ + -c defaults --override-channels \ + --variants "{'build_type': 'debug', 'freethreading': 'no'}" +``` + +`build_base.sh` already passes `--with-pydebug`, disables PGO/JIT on debug, and names `libpython${VERABI}`. Do not rewrite it unless a build proves a bug. + +## ABI tests (CI) + +Unconditional on this graph (`meta.yaml` `commands` + `run_test.py`). Do **not** rely on `PY_INTERP_DEBUG` — that is `script_env` (build), not the test env. + +```bash +python -c "import sys; assert 'd' in sys.abiflags; assert hasattr(sys, 'gettotalrefcount'); print(sys.abiflags, sys.gettotalrefcount())" +``` + +Existing `libpython${VERABI}` tests already append `d` / `td`. Linux `libpython3.so` is gated `unix and build_type == "release"` (empty upstream for debug); leave it. + +CMake `Python_FIND_ABI` already turns debug ON when `build_type == debug`. + +## gdb / lldb / ELF (local only — not a PBP gate) + +```bash +# GIL debug; FT: libpython3.15td.so +LIB=$PREFIX/lib/libpython3.15d.so +readelf -WS "$LIB" # .dynsym required; .symtab / .debug_info may be stripped +readelf -d "$LIB" +nm -D "$LIB" + +gdb -q "$PREFIX/bin/python" --batch -ex "set pagination off" \ + -ex "run -c 'import os; os.abort()'" -ex bt -ex quit +lldb -b -o "run -c 'import os; os.abort()'" -o bt -o quit "$PREFIX/bin/python" +``` + +If gdb backtraces are shallow, that is strip quality, not a missing `Py_DEBUG`. Do not expand this spike to unstrip unless it is a one-line recipe change. Use `/debug-elf` for RPATH/SONAME/arch, not for solver or graph triage. + +## Out of scope + +- numpy / pybind11 debug extension chain (follow-on after this python graph publishes to `py315-debug`; numpy-feedstock currently `skip: true # [py>=315]`) +- conda-forge `python_debug` `zip_keys` / real `channel_targets` +- pkgs/main, sanitizer/valgrind builds, commercial packaging +- Merging this PR (human-only; never pkgs/main) diff --git a/recipe/build_base.bat b/recipe/build_base.bat index cd5a6d455..4934323fb 100644 --- a/recipe/build_base.bat +++ b/recipe/build_base.bat @@ -54,7 +54,7 @@ if "%DEBUG_C%"=="yes" ( :: AP doesn't support PGO atm? set PGO= -if "%PY_GIL_DISABLED%" == "yes" ( +if "%PY_FREETHREADING%" == "yes" ( set "FREETHREADING=--disable-gil" set "THREAD=t" set "EXE_T=%VER%t" @@ -75,6 +75,13 @@ set TCLTK_MSBUILD_PROPS="/p:TclVersion=%tk%" "/p:TkVersion=%tk%" cd PCbuild +:: CF (skipped): CONDA_BUILD_CROSS_COMPILATION host-vs-build extra MSBuild. +:: AR native workers only; ARCH/BUILD_PATH above, not CF HOST_DIR/BUILD_DIR. +:: if "%CONDA_BUILD_CROSS_COMPILATION%" == "1" ( +:: set LIBRARY_PREFIX=%BUILD_PREFIX%\\Library +:: call build.bat %CONFIG% %FREETHREADING% -m -e -v -p %BUILD_PLATFORM% %TCLTK_MSBUILD_PROPS% +:: ) + :: Twice because: :: error : importlib_zipimport.h updated. You will need to rebuild pythoncore to see the changes. call build.bat %PGO% %CONFIG% %FREETHREADING% -m -e -v -p %PLATFORM% %TCLTK_MSBUILD_PROPS% @@ -84,7 +91,7 @@ if errorlevel 1 exit 1 cd .. :: Populate the root package directory -for %%x in (python%VERNODOTS%%THREAD%%_D%.dll python3%THREAD%%_D%.dll python%EXE_T%%_D%.exe pythonw%EXE_T%%_D%.exe) do ( +for %%x in (python%VERNODOTS%%THREAD%%_D%.dll python3%THREAD%%_D%.dll python%EXE_T%%_D%.exe pythonw%EXE_T%%_D%.exe python%_D%.exe) do ( if exist %SRC_DIR%\PCbuild\%BUILD_PATH%\%%x ( copy /Y %SRC_DIR%\PCbuild\%BUILD_PATH%\%%x %PREFIX% ) else ( @@ -160,6 +167,8 @@ for %%x in (lib libs) do ( if errorlevel 1 exit 1 if exist %SRC_DIR%\PCbuild\%BUILD_PATH%\_tkinter%_D%.lib copy /Y %SRC_DIR%\PCbuild\%BUILD_PATH%\_tkinter%_D%.lib %PREFIX%\%%x\ if errorlevel 1 exit 1 + if exist %SRC_DIR%\PCbuild\%BUILD_PATH%\_tkinter%THREAD%%_D%.lib copy /Y %SRC_DIR%\PCbuild\%BUILD_PATH%\_tkinter%THREAD%%_D%.lib %PREFIX%\%%x\ + if errorlevel 1 exit 1 ) @@ -173,6 +182,9 @@ if errorlevel 1 exit 1 if exist %SRC_DIR%\PCbuild\%BUILD_PATH%\venvlauncher%THREAD%%_D%.exe ( @rem We did copy pythonw.exe until 3.12 but starting with 3.13 we seem to need the latter. Should we omit the first? copy /Y %SRC_DIR%\PCbuild\%BUILD_PATH%\venvlauncher%THREAD%%_D%.exe %PREFIX%\lib\python\venv\scripts\nt\python.exe + copy /Y %SRC_DIR%\PCbuild\%BUILD_PATH%\venvlauncher%THREAD%%_D%.exe %PREFIX%\lib\python\venv\scripts\nt\venvlauncher.exe + :: FT venv copies venvlaunchert.exe when python.exe does not end in _d (CPython 3.15 Lib/venv). + copy /Y %SRC_DIR%\PCbuild\%BUILD_PATH%\venvlauncher%THREAD%%_D%.exe %PREFIX%\lib\python\venv\scripts\nt\venvlauncher%THREAD%.exe copy /Y %SRC_DIR%\PCbuild\%BUILD_PATH%\venvlauncher%THREAD%%_D%.exe %PREFIX%\lib\python\venv\scripts\nt\venvlauncher%THREAD%%_D%.exe ) else ( echo "WARNING :: %SRC_DIR%\PCbuild\%BUILD_PATH%\venvlauncher%THREAD%%_D%.exe does not exist" @@ -181,6 +193,8 @@ if exist %SRC_DIR%\PCbuild\%BUILD_PATH%\venvlauncher%THREAD%%_D%.exe ( if exist %SRC_DIR%\PCbuild\%BUILD_PATH%\venvwlauncher%THREAD%%_D%.exe ( @rem We did copy pythonw.exe until 3.12 but starting with 3.13 we seem to need the latter. Should we omit the first? copy /Y %SRC_DIR%\PCbuild\%BUILD_PATH%\venvwlauncher%THREAD%%_D%.exe %PREFIX%\lib\python\venv\scripts\nt\pythonw.exe + copy /Y %SRC_DIR%\PCbuild\%BUILD_PATH%\venvwlauncher%THREAD%%_D%.exe %PREFIX%\lib\python\venv\scripts\nt\venvwlauncher.exe + copy /Y %SRC_DIR%\PCbuild\%BUILD_PATH%\venvwlauncher%THREAD%%_D%.exe %PREFIX%\lib\python\venv\scripts\nt\venvwlauncher%THREAD%.exe copy /Y %SRC_DIR%\PCbuild\%BUILD_PATH%\venvwlauncher%THREAD%%_D%.exe %PREFIX%\lib\python\venv\scripts\nt\venvwlauncher%THREAD%%_D%.exe ) else ( echo "WARNING :: %SRC_DIR%\PCbuild\%BUILD_PATH%\venvwlauncher%THREAD%%_D%.exe does not exist" @@ -200,8 +214,11 @@ move %PREFIX%\lib\python\test_keep %PREFIX%\lib\python\test if errorlevel 1 exit 1 :: We need our Python to be found! -if "%_D%" neq "" copy %PREFIX%\python%_D%.exe %PREFIX%\python.exe -if "%EXE_T%" neq "" copy %PREFIX%\python%EXE_T%.exe %PREFIX%\python.exe +:: GIL debug: python_d.exe; FT debug: python3.15t_d.exe; skip when already python.exe. +if not "%EXE_T%%_D%"=="" ( + copy /Y %PREFIX%\python%EXE_T%%_D%.exe %PREFIX%\python.exe + if errorlevel 1 exit 1 +) set "PYTHON=%PREFIX%\python.exe" :: bytecode compile the standard library diff --git a/recipe/build_base.sh b/recipe/build_base.sh index b95b1a704..fc3cb174a 100644 --- a/recipe/build_base.sh +++ b/recipe/build_base.sh @@ -7,9 +7,12 @@ cd ${SRC_DIR} cp $BUILD_PREFIX/share/libtool/build-aux/config.* . # https://docs.python.org/3.14/using/configure.html#cmdoption-enable-experimental-jit -# says we want a python>=3.11 to build python. -if [[ ! -d ${SRC_DIR}/python-bin ]]; then - CONDA_SUBDIR=$build_platform conda create -p ${SRC_DIR}/python-bin "python>=3.11" -c main --override-channels --yes --quiet +# says we want a python>=3.11 to build python. Prefer the build env (PBP is 3.14) +# so we do not hit conda.anaconda.org/main HTTP 403 on aarch64 workers. +if python -c 'import sys; raise SystemExit(0 if sys.version_info >= (3, 11) else 1)' 2>/dev/null; then + : +elif [[ ! -d ${SRC_DIR}/python-bin ]]; then + CONDA_SUBDIR=$build_platform conda create -p ${SRC_DIR}/python-bin "python>=3.11" -c defaults --override-channels --yes --quiet export PATH=${SRC_DIR}/python-bin/bin:${PATH} fi @@ -35,24 +38,23 @@ _buildd_shared=build-shared _ENABLE_SHARED=--enable-shared # We *still* build a shared lib here for non-static embedded use cases _DISABLE_SHARED=--disable-shared -# Hack to allow easily comparing static vs shared interpreter performance -# .. hack because we just build it shared in both the build-static and -# build-shared directories. -# Yes this hack is a bit confusing, sorry about that. -if [[ ${PY_INTERP_LINKAGE_NATURE} == shared ]]; then - _DISABLE_SHARED=--enable-shared - _ENABLE_SHARED=--enable-shared -fi # For debugging builds, set this to no to disable profile-guided optimization -if [[ ${DEBUG_C} == yes ]]; then +if [[ ${PY_INTERP_DEBUG} == yes || ${DEBUG_C} == yes ]]; then _OPTIMIZED=no else _OPTIMIZED=yes fi +# CF (skipped): debug builds on conda-forge main must not ship; AR has no python_debug channel. +# if [[ ${PY_INTERP_DEBUG} == yes ]]; then +# if [[ "$CI" != "" && "$channel_targets" == "conda-forge main" ]]; then +# exit 1 +# fi +# fi + declare -a _dbg_opts -if [[ ${DEBUG_PY} == yes ]]; then +if [[ ${PY_INTERP_DEBUG} == yes ]]; then # This Python will not be usable with non-debug Python modules. _dbg_opts+=(--with-pydebug) DBG=d @@ -60,15 +62,16 @@ else DBG= fi -if [[ ${PY_GIL_DISABLED} == yes ]]; then +if [[ ${PY_FREETHREADING} == yes ]]; then # This Python will not be usable with non-free threading Python modules. THREAD=t else THREAD= fi -ABIFLAGS=${DBG} +ABIFLAGS=${DBG}${THREAD} VERABI=${VER}${THREAD}${DBG} +VERABI_NO_DBG=${VER}${THREAD} # Make sure the "python" value in conda_build_config.yaml is up to date. test "${PY_VER}" = "${VER}" @@ -107,6 +110,12 @@ if [[ ${_OPTIMIZED} = yes ]]; then CXXFLAGS=$(echo "${CXXFLAGS}" | sed "s/-O2/-O3/g") fi +if [[ ${PY_INTERP_DEBUG} == yes ]]; then + CPPFLAGS=$(echo "${CPPFLAGS}" | sed "s/-O2/-O0/g") + CFLAGS=$(echo "${CFLAGS}" | sed "s/-O2/-O0/g") + CXXFLAGS=$(echo "${CXXFLAGS}" | sed "s/-O2/-O0/g") +fi + if [[ "$target_platform" == linux-* ]]; then cp ${PREFIX}/include/uuid/uuid.h ${PREFIX}/include/uuid.h fi @@ -170,6 +179,7 @@ if [[ "${CONDA_BUILD_CROSS_COMPILATION}" == "1" ]]; then echo "ac_cv_file__dev_ptc=no" >> config.site echo "ac_cv_pthread=yes" >> config.site echo "ac_cv_little_endian_double=yes" >> config.site + # CF also lists linux-riscv64 here; AR does not build it. if [[ "${target_platform}" == "osx-arm64" || "${target_platform}" == "linux-ppc64le" || "${target_platform}" == "linux-aarch64" ]]; then echo "ac_cv_aligned_required=no" >> config.site echo "ac_cv_pthread_is_default=yes" >> config.site @@ -207,6 +217,7 @@ if [[ ${target_platform} == osx-64 ]]; then # TODO: check with LLVM 12 if the following hack is needed. # https://reviews.llvm.org/D76461 may have fixed the need for the following hack. echo '#!/bin/bash' > $BUILD_PREFIX/bin/$HOST-llvm-ar + # AR: versioned llvm-ar (CF uses unversioned llvm-ar → LLVM 20 Abort). echo "$BUILD_PREFIX/bin/llvm-ar-$LLVM_VER --format=darwin" '"$@"' >> $BUILD_PREFIX/bin/$HOST-llvm-ar chmod +x $BUILD_PREFIX/bin/$HOST-llvm-ar export ARCHFLAGS="-arch x86_64" @@ -253,14 +264,13 @@ _common_configure_args+=(--with-tzpath=${PREFIX}/share/zoneinfo) _common_configure_args+=(--with-computed-gotos) _common_configure_args+=(--with-system-expat) _common_configure_args+=(--enable-loadable-sqlite-extensions) -# TODO: Remove this comment in PR#256 # Dropped dead tcltk configure flags — log-confirmed unused in 3.15.0rc2: # configure: WARNING: unrecognized options: --with-tcltk-includes, --with-tcltk-libs # _tkinter already comes from pkg-config. _common_configure_args+=(--with-platlibdir=lib) _common_configure_args+=(--with-system-libmpdec=yes) -if [[ "${DEBUG_PY}" == "yes" || "${target_platform}" != *"64" || ${PY_GIL_DISABLED} == yes ]]; then +if [[ "${PY_INTERP_DEBUG}" == "yes" || "${target_platform}" != *"-64" || ${PY_FREETHREADING} == yes ]]; then _common_configure_args+=(--enable-experimental-jit=no) else _common_configure_args+=(--enable-experimental-jit=yes-off) @@ -271,7 +281,7 @@ if [[ "${target_platform}" == osx-* ]]; then _common_configure_args+=(--with-tail-call-interp) fi -if [[ ${PY_GIL_DISABLED} == yes ]]; then +if [[ ${PY_FREETHREADING} == yes ]]; then _common_configure_args+=(--disable-gil) fi @@ -336,29 +346,16 @@ pushd ${_buildd_static} ${_DISABLE_SHARED} "${_PROFILE_TASK[@]}" popd -if [[ "${CI}" == "travis" ]]; then - # Travis has issues with long logs - make -j${CPU_COUNT} -C ${_buildd_static} \ - EXTRA_CFLAGS="${EXTRA_CFLAGS}" \ - ${_MAKE_TARGET} "${_PROFILE_TASK[@]}" 2>&1 >make-static.log -else - make -j${CPU_COUNT} -C ${_buildd_static} \ - EXTRA_CFLAGS="${EXTRA_CFLAGS}" \ - ${_MAKE_TARGET} "${_PROFILE_TASK[@]}" 2>&1 | tee make-static.log -fi +make -j${CPU_COUNT} -C ${_buildd_static} \ + EXTRA_CFLAGS="${EXTRA_CFLAGS}" \ + ${_MAKE_TARGET} "${_PROFILE_TASK[@]}" 2>&1 | tee make-static.log if rg "Failed to build these modules" make-static.log; then echo "(static) :: Failed to build some modules, check the log" exit 1 fi -if [[ "${CI}" == "travis" ]]; then - # Travis has issues with long logs - make -j${CPU_COUNT} -C ${_buildd_shared} \ - EXTRA_CFLAGS="${EXTRA_CFLAGS}" 2>&1 >make-shared.log -else - make -j${CPU_COUNT} -C ${_buildd_shared} \ - EXTRA_CFLAGS="${EXTRA_CFLAGS}" 2>&1 | tee make-shared.log -fi +make -j${CPU_COUNT} -C ${_buildd_shared} \ + EXTRA_CFLAGS="${EXTRA_CFLAGS}" 2>&1 | tee make-shared.log if rg "Failed to build these modules" make-shared.log; then echo "(shared) :: Failed to build some modules, check the log" exit 1 @@ -367,7 +364,7 @@ fi # build a static library with PIC objects and without LTO/PGO make -j${CPU_COUNT} -C ${_buildd_shared} \ EXTRA_CFLAGS="${EXTRA_CFLAGS}" \ - LIBRARY=libpython${VERABI}-pic.a libpython${VERABI}-pic.a + LIBRARY=libpython${VERABI_NO_DBG}-pic.a libpython${VERABI_NO_DBG}-pic.a make -C ${_buildd_static} install @@ -386,46 +383,23 @@ if [[ ${_OPTIMIZED} == yes ]]; then _FLAGS_REPLACE+=("") done fi -# Install the shared library (for people who embed Python only, e.g. GDB). -# Linking module extensions to this on Linux is redundant (but harmless). -# Linking module extensions to this on Darwin is harmful (multiply defined symbols). -shopt -s extglob -cp -pf ${_buildd_shared}/libpython*${SHLIB_EXT}!(.lto) ${PREFIX}/lib/ -shopt -u extglob -if [[ ${target_platform} =~ .*linux.* ]]; then - ln -sf ${PREFIX}/lib/libpython${VERABI}${SHLIB_EXT}.1.0 ${PREFIX}/lib/libpython${VERABI}${SHLIB_EXT} -fi - -# create libpython3.dylib; linux gets libpython3.so from upstream's Makefile, -# macOS has no upstream rule — build the stable-ABI re-export dylib ourselves -# (same as CF install_shared.sh). Release only, matching linux. -if [[ "$target_platform" == osx-* && ${PY_INTERP_DEBUG} == no ]]; then - # need to filter out windows-specific symbols & PyOS_CheckStack from - # https://github.com/python/cpython/blob/main/Doc/data/stable_abi.dat - awk -F',' ' - ($1 == "func" || $1 == "data") && - $4 != "on Windows" && - $2 != "PyOS_CheckStack" { - print "_" $2 - } - ' ${SRC_DIR}/Doc/data/stable_abi.dat > ${_buildd_shared}/stable_abi_exports.txt - - $CC -dynamiclib \ - -install_name @rpath/libpython3.dylib \ - -compatibility_version 3.0 -current_version ${VER}.0 \ - -Wl,-reexport_library,${PREFIX}/lib/libpython${VERABI}.dylib \ - -Wl,-exported_symbols_list,${_buildd_shared}/stable_abi_exports.txt \ - -o ${PREFIX}/lib/libpython3.dylib -fi - -# AR: keep sysconfig from the *static* build (same as CF install_base.sh). -# A shared-build sysconfig made python3-config --embed emit -lpython3.15, so -# libpython-static tests linked the dylib and dyld aborted on osx-arm64. -SYSCONFIG=$(find ${_buildd_static}/$(cat ${_buildd_static}/pybuilddir.txt) -name "_sysconfigdata*.py" -print0) +# Use sysconfig from the shared build so extensions prefer .so (CF #565). +# Shared libpython*.so/.dylib is installed by the libpython output, not here. +BUILD_DIR=$(< ${_buildd_shared}/pybuilddir.txt) +SYSCONFIG=$(find ${_buildd_shared}/${BUILD_DIR} -name "_sysconfigdata*.py" -print0) cat ${SYSCONFIG} | ${SYS_PYTHON} "${RECIPE_DIR}"/replace-word-pairs.py \ "${_FLAGS_REPLACE[@]}" \ - > ${PREFIX}/lib/python${VERABI}/$(basename ${SYSCONFIG}) -MAKEFILE=$(find ${PREFIX}/lib/python${VERABI}/ -path "*config-*/Makefile" -print0) + > ${PREFIX}/lib/python${VERABI_NO_DBG}/$(basename ${SYSCONFIG}) +# CF copies build-details.json from the shared build (#565). Native only: +# AR does not cross-compile (CONDA_BUILD_CROSS_COMPILATION is never 1). +# if [[ "${CONDA_BUILD_CROSS_COMPILATION}" == "1" ]]; then +# # build-details.json is incorrect when cross-compiling: +# # https://github.com/python/cpython/issues/136267 +# ... +# fi +BUILD_DETAILS=${_buildd_shared}/${BUILD_DIR}/build-details.json +cp ${BUILD_DETAILS} ${PREFIX}/lib/python${VERABI_NO_DBG}/ +MAKEFILE=$(find ${PREFIX}/lib/python${VERABI_NO_DBG}/ -path "*config-*/Makefile" -print0) cp ${MAKEFILE} /tmp/Makefile-$$ cat /tmp/Makefile-$$ | ${SYS_PYTHON} "${RECIPE_DIR}"/replace-word-pairs.py \ "${_FLAGS_REPLACE[@]}" \ @@ -447,7 +421,7 @@ ln -s ${PREFIX}/bin/pydoc${VER} ${PREFIX}/bin/pydoc # Remove test data to save space # Though keep `support` as some things use that. # TODO :: Make a subpackage for this once we implement multi-level testing. -pushd ${PREFIX}/lib/python${VERABI} +pushd ${PREFIX}/lib/python${VERABI_NO_DBG} mkdir test_keep mv test/__init__.py test/support test/test_support* test/test_script_helper* test_keep/ rm -rf test */test @@ -460,7 +434,7 @@ pushd ${PREFIX} chmod +w lib/libpython${VERABI}.a ${STRIP} -S lib/libpython${VERABI}.a fi - CONFIG_LIBPYTHON=$(find lib/python${VERABI}/config-${VERABI}* -name "libpython${VERABI}.a") + CONFIG_LIBPYTHON=$(find lib/python${VERABI_NO_DBG}/config-${VERABI}* -name "libpython${VERABI}.a") if [[ -f lib/libpython${VERABI}.a ]] && [[ -f ${CONFIG_LIBPYTHON} ]]; then chmod +w ${CONFIG_LIBPYTHON} rm ${CONFIG_LIBPYTHON} @@ -470,7 +444,7 @@ popd # Copy sysconfig that gets recorded to a non-default name # using the new compilers with python will require setting _PYTHON_SYSCONFIGDATA_NAME # to the name of this file (minus the .py extension) -pushd "${PREFIX}"/lib/python${VERABI} +pushd "${PREFIX}"/lib/python${VERABI_NO_DBG} # On Python 3.5 _sysconfigdata.py was getting copied in here and compiled for some reason. # This breaks our attempt to find the right one as recorded_name. find lib-dynload -name "_sysconfigdata*.py*" -exec rm {} \; @@ -527,6 +501,7 @@ pushd "${PREFIX}"/lib/python${VERABI} popd if [[ ${HOST} =~ .*linux.* ]]; then + # AR: historical path (CF uses share/python_compiler_compat). mkdir -p ${PREFIX}/compiler_compat ln -s ${PREFIX}/bin/${HOST}-ld ${PREFIX}/compiler_compat/ld echo "Files in this folder are to enhance backwards compatibility of anaconda software with older compilers." > ${PREFIX}/compiler_compat/README @@ -535,6 +510,13 @@ fi python -c "import compileall,os;compileall.compile_dir(os.environ['PREFIX'])" rm ${PREFIX}/lib/libpython${VERABI}.a + +if [[ ${PY_INTERP_DEBUG} == yes ]]; then + rm ${PREFIX}/bin/python${VER} + ln -s ${PREFIX}/bin/python${VERABI} ${PREFIX}/bin/python${VER} + ln -s ${PREFIX}/include/python${VERABI} ${PREFIX}/include/python${VER} +fi + if [[ "$target_platform" == linux-* ]]; then rm ${PREFIX}/include/uuid.h fi @@ -552,7 +534,7 @@ fi # /lib/python3.13t/site-packages. # Note that these directories are not added to sys.path if they do not exist. SP_DIR="${PREFIX}/lib/python${PY_VER}${THREAD}/site-packages" -if [[ ${PY_GIL_DISABLED} == yes ]]; then +if [[ ${PY_FREETHREADING} == yes ]]; then echo "${PREFIX}/lib/python${PY_VER}/site-packages" >> $SP_DIR/conda-site.pth fi # Do not add ${PREFIX}/lib/python/site-packages (CF experimental ABI3/noarch diff --git a/recipe/build_static.bat b/recipe/build_static.bat deleted file mode 100644 index e69de29bb..000000000 diff --git a/recipe/build_static.sh b/recipe/build_static.sh deleted file mode 100644 index 9b5f39083..000000000 --- a/recipe/build_static.sh +++ /dev/null @@ -1,46 +0,0 @@ -#!/bin/bash -set -ex - -cd ${SRC_DIR} - -_buildd_static=build-static -_buildd_shared=build-shared -if [[ ${DEBUG_PY} == yes ]]; then - DBG=d -else - DBG= -fi -if [[ ${PY_GIL_DISABLED} == yes ]]; then - THREAD=t -else - THREAD= -fi -VER=${PKG_VERSION%.*} -VERABI=${VER}${DBG}${THREAD} - -case "$target_platform" in - linux-64) - OLD_HOST=$(echo ${HOST} | sed -e 's/-conda_cos6//g') - OLD_HOST=$(echo ${OLD_HOST} | sed -e 's/-conda_cos7//g') - OLD_HOST=$(echo ${OLD_HOST} | sed -e 's/-conda//g') - ;; - linux-*) - OLD_HOST=$(echo ${HOST} | sed -e 's/-conda_cos7//g') - OLD_HOST=$(echo ${OLD_HOST} | sed -e 's/-conda//g') - ;; - *) - OLD_HOST=$HOST - ;; -esac - -cp -pf ${_buildd_static}/libpython${VERABI}.a ${PREFIX}/lib/libpython${VERABI}.a -if [[ ${HOST} =~ .*linux.* ]]; then - pushd ${PREFIX}/lib/python${VERABI}/config-${VERABI}-${OLD_HOST} -elif [[ ${HOST} =~ .*darwin.* ]]; then - pushd ${PREFIX}/lib/python${VERABI}/config-${VERABI}-darwin -fi -ln -s ../../libpython${VERABI}.a libpython${VERABI}.a -popd -# If the LTO info in the normal lib is problematic (using different compilers for example -# we also provide a 'nolto' version). -cp -pf ${_buildd_shared}/libpython${VERABI}-pic.a ${PREFIX}/lib/libpython${VERABI}.nolto.a diff --git a/recipe/conda_build_config.yaml b/recipe/conda_build_config.yaml index 5e998d424..4ad3dc29d 100644 --- a/recipe/conda_build_config.yaml +++ b/recipe/conda_build_config.yaml @@ -14,10 +14,18 @@ python_impl: # Override aggregate global tk 8.6 — build/test against tk 9 from pkgs/main tk: - '9.0' -gil_type: - - normal - # Will be enabled as part of https://anaconda.atlassian.net/browse/PKG-5855 - - disabled # [not (s390x or (osx and x86_64))] +# Whether to build free-threaded Python (3.13+) +freethreading: + - yes + - no + +# Debug-only graph → ad-testing/label/py315-debug (PKG-2624 spike). +# Unix: --with-pydebug. Windows: PCbuild -d / python3XX_d.lib. +# Do not zip with channel_targets (F94): PBP upload_channels is graph-wide. +# Restore `release` and retarget abs.yaml before any pkgs/main / #256-style +# py315 upload — debug must not ship on the main channel. +build_type: + - debug # Override aggregate global clang 20 on osx. CPython 3.15 JIT requires LLVM 21 # (Tools/jit/_llvm.py); mixing compiler('c')=20 with clang-21/llvm-tools-21 diff --git a/recipe/install_shared.sh b/recipe/install_shared.sh new file mode 100755 index 000000000..5dafbd13f --- /dev/null +++ b/recipe/install_shared.sh @@ -0,0 +1,53 @@ +#!/bin/bash +set -ex + +_buildd_shared=build-shared +if [[ ${PY_INTERP_DEBUG} == yes ]]; then + DBG=d +else + DBG= +fi +if [[ ${PY_FREETHREADING} == yes ]]; then + # This Python will not be usable with non-free threading Python modules. + THREAD=t +else + THREAD= +fi + +VER=${PKG_VERSION%.*} +ABIFLAGS=${DBG}${THREAD} +VERABI=${VER}${THREAD}${DBG} +VERABI_NO_DBG=${VER}${THREAD} + +# Install the shared library (for people who embed Python only, e.g. GDB). +# Linking module extensions to this on Linux is redundant (but harmless). +# Linking module extensions to this on Darwin is harmful (multiply defined symbols). +shopt -s extglob +cp -pf ${_buildd_shared}/libpython*${SHLIB_EXT}!(.lto) ${PREFIX}/lib/ +shopt -u extglob +if [[ ${target_platform} =~ .*linux.* ]]; then + ln -sf ${PREFIX}/lib/libpython${VERABI}${SHLIB_EXT}.1.0 ${PREFIX}/lib/libpython${VERABI}${SHLIB_EXT} +fi +if [[ ${PY_INTERP_DEBUG} == yes ]]; then + ln -s ${PREFIX}/lib/libpython${VERABI}${SHLIB_EXT} ${PREFIX}/lib/libpython${VERABI_NO_DBG}${SHLIB_EXT} +fi + +# create libpython3.dylib; linux builds only add this in release mode, do the same on osx +if [[ "$target_platform" == osx-* && ${PY_INTERP_DEBUG} == no ]]; then + # need to filter out windows-specific symbols & PyOS_CheckStack from + # https://github.com/python/cpython/blob/main/Doc/data/stable_abi.dat + awk -F',' ' + ($1 == "func" || $1 == "data") && + $4 != "on Windows" && + $2 != "PyOS_CheckStack" { + print "_" $2 + } + ' Doc/data/stable_abi.dat > stable_abi_exports.txt + + $CC -dynamiclib \ + -install_name @rpath/libpython3.dylib \ + -compatibility_version 3.0 -current_version ${PY_VER}.0 \ + -Wl,-reexport_library,${PREFIX}/lib/libpython${VERABI}.dylib \ + -Wl,-exported_symbols_list,stable_abi_exports.txt \ + -o ${PREFIX}/lib/libpython3.dylib +fi diff --git a/recipe/install_static.sh b/recipe/install_static.sh new file mode 100755 index 000000000..f77243733 --- /dev/null +++ b/recipe/install_static.sh @@ -0,0 +1,33 @@ +#!/bin/bash +set -ex + +_buildd_static=build-static +_buildd_shared=build-shared +if [[ ${PY_INTERP_DEBUG} == yes ]]; then + DBG=d +else + DBG= +fi +if [[ ${PY_FREETHREADING} == yes ]]; then + # This Python will not be usable with non-free threading Python modules. + THREAD=t +else + THREAD= +fi + +VER=${PKG_VERSION%.*} +ABIFLAGS=${DBG}${THREAD} +VERABI=${VER}${THREAD}${DBG} +VERABI_NO_DBG=${VER}${THREAD} + +cp -pf ${_buildd_static}/libpython${VERABI}.a ${PREFIX}/lib/libpython${VERABI}.a +if [[ ${HOST} =~ .*linux.* ]]; then + pushd ${PREFIX}/lib/python${VERABI_NO_DBG}/config-${VERABI}-${HOST/-conda/} +elif [[ ${HOST} =~ .*darwin.* ]]; then + pushd ${PREFIX}/lib/python${VERABI_NO_DBG}/config-${VERABI}-darwin +fi +ln -s ../../libpython${VERABI}.a libpython${VERABI}.a +popd +# If the LTO info in the normal lib is problematic (using different compilers for example +# we also provide a 'nolto' version). +cp -pf ${_buildd_shared}/libpython${VERABI_NO_DBG}-pic.a ${PREFIX}/lib/libpython${VERABI_NO_DBG}.nolto.a diff --git a/recipe/meta.yaml b/recipe/meta.yaml index cbb252aa1..e6ce6ccf4 100644 --- a/recipe/meta.yaml +++ b/recipe/meta.yaml @@ -4,65 +4,76 @@ {% set ver2 = '.'.join(version.split('.')[0:2]) %} {% set ver2nd = ''.join(version.split('.')[0:2]) %} {% set ver3nd = ''.join(version.split('.')[0:3]) %} -{% set build_number = 0 %} +{% set build_number = 1 %} + {% set channel_targets = ('abc', 'def') %} -# this is just for the initial build, to break dependencies with python -> pip -> libpython-static -{% set bootstrap = "false" %} # this makes the linter happy {% set channel_targets = channel_targets or 'conda-forge main' %} # Sanitize build system env. var tweak parameters # (passed to the build scripts via script_env). -{% set from_source_control = os.environ.get('CONDA_BUILD_FROM_SOURCE_CONTROL', '') %} -{% if from_source_control == '' or from_source_control == 'no' %} - {% set from_source_control = 'no' %} -{% else %} - {% set from_source_control = 'yes' %} +{% if freethreading is not defined %} + {% set freethreading = "yes" %} {% endif %} -{% set linkage_nature = os.environ.get('PY_INTERP_LINKAGE_NATURE', '') %} -{% if linkage_nature != '' %} - {% set linkage_nature = "_" ~ linkage_nature %} -{% endif %} -{% if linkage_nature == 'shared' %} - {% set linkage_nature_env = 'shared' %} + +{% set abi_tag = "cp" + ver2nd %} +# freethreading, set using conda_build_config.yaml +{% if freethreading == "yes" %} + {% set abi_tag = abi_tag + "t" %} {% else %} - {% set linkage_nature_env = 'static' %} + {% set build_number = build_number + 100 %} {% endif %} -{% set dbg_abi = "" %} -{% set debug = os.environ.get('PY_INTERP_DEBUG', '') %} -{% if debug != '' and debug != 'no' %} + +# build_type, set using conda_build_config.yaml +{% if build_type == "debug" %} {% set py_interp_debug = "yes" %} - {% set debug = "_dbg" %} - {% set dbg_abi = "d" %} # [win] + {% set debug = "_debug" %} {% else %} {% set py_interp_debug = "no" %} + {% set debug = "" %} +{% endif %} +{% set win_lib_sfx = "t" if freethreading == "yes" else "" %} +{% if py_interp_debug == "yes" %} + {% set win_lib_sfx = win_lib_sfx ~ "_d" %} {% endif %} -# no-GIL, set using conda_build_config.yaml -{% if gil_type is defined and gil_type == "disabled" %} - {% set py_gil_disabled = "yes" %} - {% set abi_suffix = "t" %} +{% if win %} + {% set site_packages_path = "lib/python/site-packages" %} +{% elif freethreading == "yes" %} + {% set site_packages_path = "lib/python" + ver2 + "t/site-packages" %} {% else %} - {% set py_gil_disabled = "no" %} - {% set abi_suffix = "" %} - {% set build_number = build_number + 100 %} + {% set site_packages_path = "lib/python" + ver2 + "/site-packages" %} +{% endif %} + +# CMake needs to be told that the debug/freethreading ABI is fine; the option takes a 4-tuple, +# Python_FIND_ABI=;;;; for details see +# https://cmake.org/cmake/help/latest/module/FindPython.html#hints +{% set cmake_test_args=" -DPython_FIND_ABI=\"" ~ ("ON" if build_type == "debug" else "OFF") ~ ";ANY;ANY;" ~ ("ON" if freethreading == "yes" else "OFF") ~ "\"" %} +{% if unix %} + # for CMake invocations during testing, we need to avoid accidentally picking up python versions from + # the image, but we don't want to use `${CMAKE_ARGS}` from the compiler activation, because that contains + # `CMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER`; while that's useful for building (i.e. don't look in `host:`, + # because we want to use binaries from `build:`), we want the exact opposite when running tests. + {% set cmake_test_args=cmake_test_args ~ " -DCMAKE_POLICY_DEFAULT_CMP0094=NEW -DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=ONLY -DCMAKE_FIND_ROOT_PATH=${PREFIX}" %} +{% elif win %} + {% set cmake_test_args=cmake_test_args ~ " -DPython_ROOT_DIR=%PREFIX% -DPython_EXECUTABLE=%PREFIX%\\python.exe -DPython_LIBRARY=%PREFIX%\\libs\\python" ~ ver2nd ~ win_lib_sfx ~ ".lib -DPython_INCLUDE_DIR=%PREFIX%\\include\\python -DPython_FIND_REGISTRY=NEVER -DPython_FIND_STRATEGY=LOCATION" %} +{% endif %} + +# extremely noisy; only use if needing to debug how CMake finds our libraries +{% set cmake_debug = "" %} +{% if False %} + {% set cmake_debug = "--debug-find --trace --debug-output --debug-trycompile" %} {% endif %} -{% set abi_tag = "cp" + ver2nd + abi_suffix %} package: name: python-split version: {{ version }}{{ dev }} source: -{% if from_source_control == 'yes' %} - - git_url: https://github.com/python/CPython.git - git_tag: v{{ version }}{{ dev }} -{% else %} - url: https://www.python.org/ftp/python/{{ version }}/Python-{{ version }}{{ dev }}.tar.xz - # md5 from: https://www.python.org/downloads/release/python-{{ ver3nd }}/ + # sha256 from: https://www.python.org/downloads/release/python-{{ ver3nd }}{{ dev }}/ sha256: 8d93af5eaaaea5adfd41bd786a7ba3f03f2ad1ab57c6a65e0b963deab91d5ad7 -{% endif %} patches: - patches/0001-Win32-Change-FD_SETSIZE-from-512-to-2048.patch - patches/0002-Win32-Do-not-download-externals.patch @@ -94,22 +105,22 @@ source: - patches/0023-branding.patch - patches/0024-Do-not-define-pid_t-as-it-might-conflict-with-the-ac.patch - patches/0025-Unvendor-zstd.patch - - patches/0026-Set-Py_GIL_DISABLED-in-windows-to-match-unix.patch # [win and gil_type == "disabled"] +{% if freethreading == "yes" %} + - patches/0026-Set-Py_GIL_DISABLED-in-windows-to-match-unix.patch +{% endif %} - patches/0027-More-unixy-layout-for-windows.patch - patches/0028-Avoid-erroring-when-conda-site.pth-has-non-existent-.patch - patches/0029-Unvendor-zlib-ng.patch - - patches/0030-xxlimited-freethreading-stable-abi-link.patch # [win and gil_type == "disabled"] +{% if freethreading == "yes" %} + - patches/0030-xxlimited-freethreading-stable-abi-link.patch +{% endif %} build: number: {{ build_number }} requirements: build: -{% if from_source_control == 'yes' %} - - git -{% else %} - msys2-gcc-libs # [win] -{% endif %} outputs: - name: python @@ -126,8 +137,14 @@ outputs: # C++ only installed so CXX is defined for distutils/sysconfig. - {{ compiler('cxx') }} # [unix] # These two are just to get the headers needed for tk.h, but is unused - - xorg-libx11 # [linux] - - xorg-xorgproto # [linux] + - xorg-libx11 # [unix] + - xorg-xorgproto # [unix] +{% if py_interp_debug == "yes" %} + ignore_run_exports: # [win] + # PCbuild -d links the debug CRT, not release ucrt / vc14_runtime. + - ucrt # [win] + - vc14_runtime # [win] +{% endif %} # Disabled until verified to work correctly detect_binary_files_with_prefix: true # detect_binary_files_with_prefix: False @@ -141,30 +158,26 @@ outputs: # - lib/libpython3.dylib # [osx] # match python.org compiler standard skip: true # [win and int(float(vc)) < 14] -{% if 'conda-forge' in channel_targets %} - skip_compile_pyc: - - '*.py' # [build_platform != target_platform] -{% endif %} - string: {{ dev_ }}h{{ PKG_HASH }}_{{ PKG_BUILDNUM }}{{ linkage_nature }}{{ debug }}_{{ abi_tag }} + string: {{ dev_ }}h{{ PKG_HASH }}_{{ PKG_BUILDNUM }}{{ debug }}_{{ abi_tag }} + python_site_packages_path: {{ site_packages_path }} run_exports: noarch: - python weak: - python_abi {{ ver2 }}.* *_{{ abi_tag }} - track_features: # [gil_type == "disabled"] - - free-threading # [gil_type == "disabled"] +{% if py_interp_debug == "yes" %} + - python {{ ver2 }}.* *_debug_{{ abi_tag }} +{% endif %} +{% if freethreading == "yes" %} + track_features: + - py_freethreading +{% endif %} script_env: - - PY_INTERP_LINKAGE_NATURE={{ linkage_nature_env }} - PY_INTERP_DEBUG={{ py_interp_debug }} - - PY_GIL_DISABLED={{ py_gil_disabled }} + - PY_FREETHREADING={{ freethreading }} # Putting these here means they get emitted to build_env_setup.{sh,bat} meaning we can launch IDEs # after sourcing or calling that script without examine the contents of conda_build.{sh,bat} for # important env. vars. -{% if 'conda-forge' in channel_targets %} - - CONDA_FORGE=yes -{% else %} - - CONDA_FORGE=no -{% endif %} missing_dso_whitelist: # [win or linux] - '**/MSVCR71.dll' # [win] - '**/MSVCR80.dll' # [win] @@ -189,57 +202,50 @@ outputs: - libtool # [unix] - pkg-config # [not win] - ld_impl_{{ target_platform }} # [linux] -{% if 'conda-forge' in channel_targets %} - - binutils_impl_{{ target_platform }} # [linux] -{% endif %} - # called in build_base.sh - - sed # [unix] - - ripgrep # [not win] + # we haven't used binutils for builds of python on main + #- binutils_impl_{{ target_platform }} # [linux] # https://github.com/python/cpython/blob/main/Tools/jit/README.md#installing-llvm: # "LLVM version 21 is the officially supported version" - clang-21 # [not win] # configure script looks for llvm-ar for lto. - # llvm-tools-21 also supplies llvm-ar-21, which build_base.sh wraps as the lto archiver - # that configure picks up via $HOST-llvm-ar. + # llvm-tools-21 also supplies llvm-ar-21, which build_base.sh + # wraps as the lto archiver that configure picks up via + # $HOST-llvm-ar. - llvm-tools-21 # [not win] - clang 21.* # [win] - llvm-tools 21.* # [win] + # called in build_base.sh + - sed # [unix] + - ripgrep # [unix] host: - bzip2 {{ bzip2 }} - - sqlite {{ sqlite }} + - expat {{ expat }} + - ld_impl_{{ target_platform }} >=2.36.1 # [linux] + - libffi {{ libffi }} - xz {{ xz }} + - libmpdec-devel {{ libmpdec }} + - libuuid {{ libuuid }} # [linux] + - sqlite {{ sqlite }} + - ncurses {{ ncurses }} # [unix] + - openssl {{ openssl }} + - readline {{ readline }} # [unix] + - tk {{ tk }} + # These two are just to get the headers needed for tk.h, but is unused + - xorg-libx11 {{ xorg_libx11 }} # [unix] + - xorg-xorgproto {{ xorg_xorgproto }} # [unix] # Although we might like to use zlib-ng something else is # pulling in zlib and we get an overlinking error - zlib {{ zlib }} #- zlib-ng {{ zlib_ng }} - zstd {{ zstd }} - - openssl {{ openssl }} - - readline {{ readline }} # [not win] - - tk {{ tk }} - # These two are just to get the headers needed for tk.h, but is unused - - xorg-libx11 {{ xorg_libx11 }} # [linux] - - xorg-xorgproto {{ xorg_xorgproto }} # [linux] - - ncurses {{ ncurses }} # [unix] - - libffi {{ libffi }} - - ld_impl_{{ target_platform }} >=2.35.1 # [linux] - - libuuid {{ libuuid }} # [linux] - - libmpdec-devel {{ libmpdec }} - - expat {{ expat }} run: - - ld_impl_{{ target_platform }} >=2.35.1 # [linux] + - ld_impl_{{ target_platform }} >=2.36.1 # [linux] - tzdata - python_abi {{ ver2 }}.* *_{{ abi_tag }} -{% if 'conda-forge' in channel_targets %} - - ncurses # [unix] - run_constrained: - - python_abi {{ ver2 }}.* *_cp{{ ver2nd }} -{% endif %} test: -{% if 'conda-forge' in channel_targets %} - downstreams: - - cython - - setuptools -{% endif %} + #downstreams: + # - cython + # - setuptools requires: - ripgrep - cmake-no-system @@ -252,16 +258,25 @@ outputs: - {{ compiler('c') }} # Tried to use enable_language(C) to avoid needing this. It does not work. - {{ compiler('cxx') }} + # For tkinter, but we expect from system + - xorg-libx11 # [unix] files: - tests/cmake/* - tests/cython/* - - tests/prefix-replacement/* - run_test.py commands: - echo on # [win] - set # [win] - python -V - python3 -V # [not win] + # This graph is debug-only; do not rely on PY_INTERP_DEBUG getenv in tests. + - python -c "import sys; assert 'd' in sys.abiflags; assert hasattr(sys, 'gettotalrefcount'); print(sys.abiflags, sys.gettotalrefcount())" # [unix] + - python -c "import sys; assert hasattr(sys, 'gettotalrefcount'); print(sys.gettotalrefcount())" # [win] + - if not exist %PREFIX%\libs\python{{ ver2nd }}{% if freethreading == "yes" %}t{% endif %}_d.lib exit 1 # [win] + - if not exist %PREFIX%\libs\python3{% if freethreading == "yes" %}t{% endif %}_d.lib exit 1 # [win] +{% if freethreading != "yes" %} + - if not exist %PREFIX%\libs\_tkinter_d.lib exit 1 # [win] +{% endif %} - pydoc -h - python3-config --help # [not win] - set "PIP_NO_BUILD_ISOLATION=False" # [win] @@ -281,12 +296,11 @@ outputs: - _CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_linux_gnu python -c "import sysconfig; print(sysconfig.get_config_var('CC'))" # [linux64] # check for unreplaced @ symbols in sysconfig files, excluding e.g. '"$@".lto' - for f in ${CONDA_PREFIX}/lib/python*/_sysconfig*.py; do echo "Checking $f:"; if [[ `rg @[^@]*@ $f` ]]; then echo "FAILED ON $f"; cat $f; exit 1; fi; done # [linux64 or osx] - # AR: no separate libpython output (CF has one). Shared libpython*.so/.dylib - # ships in this package, so do not assert their absence. Static libs must not. + - test ! -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}.so # [unix] - test ! -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}.a # [unix] + - test ! -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}.so # [linux] + - test ! -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}.dylib # [osx] - test ! -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}.nolto.a # [unix] - # stable-ABI re-export dylib (upstream has no macOS rule; CF install_shared.sh parity) - - test -f ${PREFIX}/lib/libpython3.dylib # [osx] # https://github.com/conda-forge/python-feedstock/issues/384 - if exist %PREFIX%\\Scripts\\pydoc exit 1 # [win] - if exist %PREFIX%\\Scripts\\idle exit 1 # [win] @@ -294,19 +308,12 @@ outputs: - if not exist %PREFIX%\\Scripts\\idle-script.py exit 1 # [win] - if not exist %PREFIX%\\Scripts\\idle.exe exit 1 # [win] - if not exist %PREFIX%\\Scripts\\pydoc.exe exit 1 # [win] - # 0027 unixy Win layout: getpath landmarks lib\python\os.py and include\python. - if not exist %PREFIX%\\include\\python\\pyconfig.h exit 1 # [win] - - if not exist %PREFIX%\\lib\\python{{ ver2nd }}{{ abi_suffix }}.lib exit 1 # [win] - # libs\ kept for link-time compat (CFEP-27 section 6, conda-forge PR 918) - - if not exist %PREFIX%\\libs\\python{{ ver2nd }}{{ abi_suffix }}.lib exit 1 # [win] - - pushd tests - - pushd prefix-replacement # [unix] - - bash build-and-test.sh # [unix] - - popd # [unix] - - pushd cmake - - cmake -GNinja -DPY_VER={{ version }} -DEXTRA_COMPONENTS="Development.Module" --debug-find --trace --debug-output --debug-trycompile . - - popd - - popd + - pushd tests # [unix] + - pushd cmake # [unix] + - cmake -GNinja {{ cmake_test_args }} -DPY_VER={{ version }} -DEXTRA_COMPONENTS="Development.Module" {{ cmake_debug }} . # [unix] + - popd # [unix] + - popd # [unix] - python run_test.py - test ! -f default.profraw # [osx] # Test for segfault on osx-64 with libffi=3.4, see https://bugs.python.org/issue44556 @@ -314,84 +321,165 @@ outputs: # Test for wide character supported via ncursesw - TERM=xterm >/dev/null python -c "import curses; scr = curses.initscr(); curses.unget_wch('x'); assert 'x' == scr.get_wch()" # [unix] -{% if bootstrap != "true" %} + # this output is empty on windows + - name: libpython + script: install_shared.sh # [unix] + build: + number: {{ build_number }} + activate_in_script: true + ignore_run_exports: + - python_abi + ignore_run_exports_from: # [unix] + # Install-only output; CXX is not linked into libpython*.so. + - {{ compiler('cxx') }} # [unix] + script_env: + - PY_INTERP_DEBUG={{ py_interp_debug }} + - PY_FREETHREADING={{ freethreading }} + string: h{{ PKG_HASH }}_{{ PKG_BUILDNUM }}{{ debug }}_{{ abi_tag }} + run_exports: # [unix] + weak: # [unix] + - {{ pin_subpackage("libpython", min_pin="x.x", max_pin="x") }} # [unix] + requirements: + build: + - {{ compiler('c') }} + - {{ stdlib('c') }} + - {{ compiler('cxx') }} + host: + - {{ pin_subpackage('python', exact=True) }} + run: + - {{ pin_subpackage('python', exact=True) }} + test: + files: + - tests/cmake/* + - tests/prefix-replacement/* + requires: + - {{ stdlib('c') }} + - {{ compiler('c') }} + # cmake expects a C++ compiler for some reason + - {{ compiler('cxx') }} + - ripgrep + - cmake-no-system + - ninja-base + commands: + - VER=${PKG_VERSION%.*} # [not win] + - VERABI=${VER} # [not win] +{% if freethreading == "yes" %} + - VERABI=${VERABI}t # [not win] +{% endif %} + - VERABI_NO_DBG=${VERABI} # [not win] +{% if py_interp_debug == "yes" %} + - VERABI=${VERABI}d # [not win] +{% endif %} + - test -f ${PREFIX}/lib/libpython${VERABI}${SHLIB_EXT} # [unix] + - test -f ${PREFIX}/lib/libpython3${SHLIB_EXT} # [unix and build_type == "release"] + - pushd tests + - pushd prefix-replacement # [unix] + - bash build-and-test.sh # [unix] + - popd # [unix] + - pushd cmake # [unix] + - cmake -GNinja {{ cmake_test_args }} -DPY_VER={{ version }} -DEXTRA_COMPONENTS="Development.Embed" {{ cmake_debug }} . # [unix] + - popd # [unix] + - popd # [unix] + - name: libpython-static - script: build_static.sh # [unix] - script: build_static.bat # [win] + script: install_static.sh # [unix] build: number: {{ build_number }} + # hints for the CI system force_use_keys: - - gil_type + - freethreading + - build_type activate_in_script: true ignore_run_exports: - python_abi - string: h{{ PKG_HASH }}_{{ PKG_BUILDNUM }}{{ linkage_nature }}{{ debug }}_{{ abi_tag }} + ignore_run_exports_from: # [unix] + # Install-only output; CXX is not linked into libpython*.a. + - {{ compiler('cxx') }} # [unix] script_env: - - PY_GIL_DISABLED={{ py_gil_disabled }} + - PY_INTERP_DEBUG={{ py_interp_debug }} + - PY_FREETHREADING={{ freethreading }} + string: h{{ PKG_HASH }}_{{ PKG_BUILDNUM }}{{ debug }}_{{ abi_tag }} requirements: build: - {{ stdlib('c') }} - {{ compiler('c') }} - {{ compiler('cxx') }} -{% if from_source_control == 'yes' %} - - git -{% endif %} host: - {{ pin_subpackage('python', exact=True) }} run: - {{ pin_subpackage('python', exact=True) }} test: files: + - tests/cmake/* - tests/prefix-replacement/* requires: - {{ stdlib('c') }} - {{ compiler('c') }} - # Tried to use enable_language(C) to avoid needing this. It does not work. + # cmake expects a C++ compiler for some reason - {{ compiler('cxx') }} - ripgrep - + - cmake-no-system + - ninja-base commands: - - pushd tests # [unix] - - pushd prefix-replacement # [unix] - - test -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}{{ abi_suffix }}.a # [unix] - - test -f ${PREFIX}/lib/libpython${PKG_VERSION%.*}{{ abi_suffix }}.nolto.a # [unix] - - test -f ${PREFIX}/lib/python${PKG_VERSION%.*}{{ abi_suffix }}/config-${PKG_VERSION%.*}{{ abi_suffix }}-darwin/libpython${PKG_VERSION%.*}{{ abi_suffix }}.a # [osx] - - bash build-and-test.sh # [unix] - - popd # [unix] - - popd # [unix] + - VER=${PKG_VERSION%.*} # [not win] + - VERABI=${VER} # [not win] +{% if freethreading == "yes" %} + - VERABI=${VERABI}t # [not win] {% endif %} + - VERABI_NO_DBG=${VERABI} # [not win] +{% if py_interp_debug == "yes" %} + - VERABI=${VERABI}d # [not win] +{% endif %} + - test -f ${PREFIX}/lib/libpython${VERABI}.a # [unix] + - test -f ${PREFIX}/lib/libpython${VERABI_NO_DBG}.nolto.a # [unix] + - test -f ${PREFIX}/lib/python${VERABI_NO_DBG}/config-${VERABI}-darwin/libpython${VERABI}.a # [osx] + - pushd tests + - pushd prefix-replacement # [unix] + - bash build-and-test.sh # [unix] + - popd # [unix] + - pushd cmake # [unix] + - cmake -GNinja {{ cmake_test_args }} -DPY_VER={{ version }} -DEXTRA_COMPONENTS="Development.Embed" {{ cmake_debug }} . # [unix] + - popd # [unix] + - popd # [unix] + + - name: cpython + build: + noarch: generic + requirements: + - python {{ version }}{{ dev }} + - python_abi * *_{{ abi_tag }} - name: python-freethreading build: noarch: generic -{% if gil_type == "normal" %} - skip: True +{% if freethreading != "yes" %} + skip: true {% endif %} requirements: - - python {{ version }}{{ dev }}.* + - cpython {{ version }}{{ dev }} - python_abi * *_{{ abi_tag }} - name: python-gil build: noarch: generic -{% if gil_type == "disabled" %} - skip: True +{% if freethreading == "yes" %} + skip: true {% endif %} requirements: - - python {{ version }}{{ dev }}.* + - cpython {{ version }}{{ dev }} - python_abi * *_{{ abi_tag }} - name: python-jit script: install_jit_activation.sh # [not win] script: install_jit_activation.bat # [win] build: -{% if gil_type == "disabled" %} - skip: True +{% if freethreading == "yes" %} + skip: true {% endif %} skip: true # [py<313] noarch: generic requirements: - - python {{ version }}{{ dev }} + - cpython {{ version }}{{ dev }} - __unix # [unix] - __win # [win] test: @@ -414,8 +502,29 @@ about: fewer lines of code than would be possible in languages such as C++ or Java. The language provides constructs intended to enable clear programs on both a small and large scale. - doc_url: https://www.python.org/doc/versions/ - dev_url: https://devguide.python.org/ + + We provide some meta packages for convenience. + To get a CPython flavour, use: + + conda install cpython + + To get the freethreading build (i.e. without the Global Interpreter Lock - GIL): + + conda install python-freethreading + + To get the default build (i.e. with the GIL): + + conda install python-gil + + To enable the use of the experimental JIT compiler in CPython: + + conda install python-jit + + or set the environment variable PYTHON_JIT=1. Note that the JIT support + is available for x86_64 builds only. + + doc_url: https://devguide.python.org/ + dev_url: https://github.com/python/cpython extra: feedstock-name: python @@ -432,4 +541,4 @@ extra: - xhochy skip-lints: - license_file_overspecified - - parsing_jinja_failure # gil_type / channel_targets jinja; linter cannot expand this recipe + - parsing_jinja_failure # freethreading / channel_targets jinja; linter cannot expand this recipe diff --git a/recipe/run_test.py b/recipe/run_test.py index 11384abbf..d2c5b050d 100644 --- a/recipe/run_test.py +++ b/recipe/run_test.py @@ -90,10 +90,11 @@ import syslog import termios -if os.getenv('PY_INTERP_DEBUG') == 'yes': - if sys.platform != 'win32': - assert 'd' in sys.abiflags - assert 'gettotalrefcount' in dir(sys) +# This graph is debug-only (PKG-2624). Do not rely on PY_INTERP_DEBUG getenv; +# script_env does not reliably reach the test env. +if sys.platform != 'win32': + assert 'd' in sys.abiflags +assert hasattr(sys, 'gettotalrefcount') if not (armv6l or armv7l or ppc64le or osx105 or arm64): import tkinter diff --git a/recipe/run_test_jit.py b/recipe/run_test_jit.py index ae203f783..154fc1716 100644 --- a/recipe/run_test_jit.py +++ b/recipe/run_test_jit.py @@ -12,10 +12,14 @@ sys.exit(0) else: print("JIT available but not enabled", file=sys.stderr) + sys.exit(1) else: - print("JIT not available", file=sys.stderr) + # python-jit is noarch and PBP tests it against every python variant. + # configure disables JIT for debug, freethreading, and some arches; + # the metapackage is a no-op there. + print("JIT not compiled into this interpreter; skip") + sys.exit(0) else: print(f"WARNING: cannot validate JIT using Python {sys.version}", file=sys.stderr) - print(f"WARNING: validate using Python >= 3.14", file=sys.stderr) - -sys.exit(1) + print("WARNING: validate using Python >= 3.14", file=sys.stderr) + sys.exit(1) diff --git a/recipe/tests/cmake/CMakeLists.txt b/recipe/tests/cmake/CMakeLists.txt index aa7de388e..1b72e0c9b 100644 --- a/recipe/tests/cmake/CMakeLists.txt +++ b/recipe/tests/cmake/CMakeLists.txt @@ -10,7 +10,10 @@ set(PY_VER "" CACHE STRING "Python version to use") set(EXTRA_COMPONENTS "" CACHE STRING "Additional Python components to check for") # Since 3.16, CMake does not accept freethreaded interpreter by default -# anymore. Allow it explicitly. +# anymore. Allow it explicitly unless the caller passed -DPython_FIND_ABI +# (debug builds need the first tuple element ON). # https://cmake.org/cmake/help/latest/module/FindPython.html#hints -set(Python_FIND_ABI "ANY" "ANY" "ANY" "ANY") +if(NOT DEFINED Python_FIND_ABI) + set(Python_FIND_ABI "ANY" "ANY" "ANY" "ANY") +endif() find_package(Python ${PY_VER} COMPONENTS Interpreter ${EXTRA_COMPONENTS} REQUIRED) diff --git a/recipe/tests/prefix-replacement/a.c b/recipe/tests/prefix-replacement/a.c index dacc1d63e..93922c20d 100644 --- a/recipe/tests/prefix-replacement/a.c +++ b/recipe/tests/prefix-replacement/a.c @@ -9,7 +9,8 @@ main(int argc, char *argv[]) fprintf(stderr, "Fatal error: cannot decode argv[0]\n"); exit(1); } - Py_SetProgramName(program); /* optional but recommended */ + // Removed in Python 3.13 and needs to be replaced with https://docs.python.org/3.13/c-api/init_config.html#c.PyConfig.program_name + // Py_SetProgramName(program); /* optional but recommended */ Py_Initialize(); PyRun_SimpleString("from time import time,ctime\n" "print('Today is', ctime(time()))\n"); diff --git a/recipe/tests/prefix-replacement/build-and-test.sh b/recipe/tests/prefix-replacement/build-and-test.sh index 633b741a3..d710bde75 100644 --- a/recipe/tests/prefix-replacement/build-and-test.sh +++ b/recipe/tests/prefix-replacement/build-and-test.sh @@ -2,46 +2,47 @@ set -ex -if [[ "$PKG_NAME" == "libpython-static" ]]; then - # see bpo44182 for why -L${CONDA_PREFIX}/lib is added - ${CC} a.c $(python3-config --cflags) $(python3-config --embed --ldflags) -L${CONDA_PREFIX}/lib -o ${CONDA_PREFIX}/bin/embedded-python-static - if [[ "$target_platform" == linux-* ]]; then - if ${READELF} -d ${CONDA_PREFIX}/bin/embedded-python-static | rg libpython; then - echo "ERROR :: Embedded python linked to shared python library. It is expected to link to the static library." - fi - elif [[ "$target_platform" == osx-* ]]; then - if ${OTOOL} -l ${CONDA_PREFIX}/bin/embedded-python-static | rg libpython; then - echo "ERROR :: Embedded python linked to shared python library. It is expected to link to the static library." - fi - fi - ${CONDA_PREFIX}/bin/embedded-python-static - - # I thought this would prefer the shared library for Python. I was wrong: - # EMBED_LDFLAGS=$(python3-config --ldflags) - # re='^(.*)(-lpython[^ ]*)(.*)$' - # if [[ ${EMBED_LDFLAGS} =~ $re ]]; then - # EMBED_LDFLAGS="${BASH_REMATCH[1]} ${BASH_REMATCH[3]} -Wl,-Bdynamic ${BASH_REMATCH[2]}" - # fi - # ${CC} a.c $(python3-config --cflags) ${EMBED_LDFLAGS} -o ${CONDA_PREFIX}/bin/embedded-python-shared +case ${PKG_NAME} in + libpython) + # see bpo44182 for why -L${CONDA_PREFIX}/lib is added + ${CC} a.c $(python3-config --cflags) \ + $(python3-config --embed --ldflags) \ + -L${CONDA_PREFIX}/lib -Wl,-rpath,${CONDA_PREFIX}/lib \ + -o ${CONDA_PREFIX}/bin/embedded-python-shared - # Brute-force way of linking to the shared library, sorry! - rm -rf ${CONDA_PREFIX}/lib/libpython*.a -fi + if [[ "$target_platform" == linux-* ]]; then + if ! ${READELF} -d ${CONDA_PREFIX}/bin/embedded-python-shared | rg libpython; then + echo "ERROR :: Embedded python linked to static python library. We tried to force it to use the shared library." + fi + elif [[ "$target_platform" == osx-* ]]; then + if ! ${OTOOL} -l ${CONDA_PREFIX}/bin/embedded-python-shared | rg libpython; then + echo "ERROR :: Embedded python linked to static python library. We tried to force it to use the shared library." + fi + fi + ${CONDA_PREFIX}/bin/embedded-python-shared + ;; -${CC} a.c $(python3-config --cflags) \ - $(python3-config --embed --ldflags) \ - -L${CONDA_PREFIX}/lib -Wl,-rpath,${CONDA_PREFIX}/lib \ - -o ${CONDA_PREFIX}/bin/embedded-python-shared + libpython-static) + ${CC} a.c $(python3-config --cflags) \ + $(python3-config --embed --ldflags) \ + -L${CONDA_PREFIX}/lib -Wl,-rpath,${CONDA_PREFIX}/lib \ + -o ${CONDA_PREFIX}/bin/embedded-python-static + if [[ "$target_platform" == linux-* ]]; then + if ${READELF} -d ${CONDA_PREFIX}/bin/embedded-python-static | rg libpython; then + echo "ERROR :: Embedded python linked to shared python library. It is expected to link to the static library." + fi + elif [[ "$target_platform" == osx-* ]]; then + if ${OTOOL} -l ${CONDA_PREFIX}/bin/embedded-python-static | rg libpython; then + echo "ERROR :: Embedded python linked to shared python library. It is expected to link to the static library." + fi + fi + ${CONDA_PREFIX}/bin/embedded-python-static + ;; -if [[ "$target_platform" == linux-* ]]; then - if ! ${READELF} -d ${CONDA_PREFIX}/bin/embedded-python-shared | rg libpython; then - echo "ERROR :: Embedded python linked to static python library. We tried to force it to use the shared library." - fi -elif [[ "$target_platform" == osx-* ]]; then - if ! ${OTOOL} -l ${CONDA_PREFIX}/bin/embedded-python-shared | rg libpython; then - echo "ERROR :: Embedded python linked to static python library. We tried to force it to use the shared library." - fi -fi -${CONDA_PREFIX}/bin/embedded-python-shared + *) + # invalid package + exit 1 + ;; +esac set +x