From a22b806d02d7ee475a087a0036c5a83990aacd8d Mon Sep 17 00:00:00 2001 From: Anthony Shoumikhin Date: Mon, 2 Nov 2026 08:40:26 -0800 Subject: [PATCH] Move MLX, Build Presets and macOS wheel CI to macOS 15 runners (#24402) GitHub retires its macOS 14 runner images on November 2, 2026 ([announcement](https://github.com/actions/runner-images/issues/13518)). After that, jobs that ask for `macos-14-xlarge` fail, and the MLX tests, the Apple preset builds and the macOS wheel builds still do. ### What this changes These jobs now ask for `macos-15-xlarge`, like the Apple and trunk workflows. The MLX LLM jobs no longer pick a runner per model; that only existed because Gemma 4 gave wrong output on macOS 14 (#19065). The MLX jobs that build through presets now build at one less than the core count. Their presets either run an unlimited `make -j` or build one file at a time, and on macOS 15 these jobs took 55 to 90 minutes instead of about 40. The wheel smoke test now checks that the macOS wheel still targets macOS 14. It fails if the tag, or any Mach-O file or Metal library in the wheel, asks for a newer macOS. This used to come for free, because the smoke test ran on macOS 14. The macOS wheel keeps the Arm VGF backend off, as before. Its library ships only for macOS 15, so it would put code built for macOS 15.4 into a wheel that claims macOS 14. The wheel build also pins Xcode 16.4, so a runner image update cannot change the wheel. Nothing runs on macOS 14 any more, which leaves two gaps for follow-ups. The check cannot see a call to a macOS 15-only API made without an availability check, or code built for macOS 15 that is linked into a library. And MLX is no longer tested on macOS 14, where Gemma 4 gave wrong output. ### What it means for users The wheel keeps its `macosx_14_0_arm64` tag, which comes from the build settings, not from the build machine. Built with the macOS 15.5 SDK, its Core ML backend can now run multifunction and stateful (`MLState`) models on macOS 15 and newer. That code checks the OS version at run time, so macOS 14 behaves as before. ### Test plan - On macOS 15, CI built every Apple preset and the Python 3.11 macOS wheel. The wheel is tagged `macosx_14_0_arm64`, is built with Xcode 16.4, has the Metal library and no VGF code, and passes the new check. An earlier commit also built Python 3.12 to 3.15 (`ciflow/binaries`), all with the same tag. The Linux wheels still build VGF. - Every MLX job passed on macOS 15, including the ones that need secrets (`ciflow/mlx`). The capped jobs took 13 to 22 minutes. Voxtral Realtime once printed only unknown tokens on an earlier commit, then passed when rerun. - Unit tests cover the check's decision and parsers, and run the whole check on a fake wheel with `lipo` and `otool` stubbed. With the real tools, the check also fails on altered nightly wheels: a wrong tag, a library that needs macOS 14.5 or 15.0, a `flatc` that needs macOS 15.0 (also as one slice of a universal binary), and a Metal library built for macOS 15.0 or with no target. (cherry picked from commit 4ea7e9d9ba353358aec051a6096ba90567828494) --- .ci/scripts/tests/test_wheel_platform_tag.py | 150 ++++++++++++++++++- .ci/scripts/wheel/envvar_macos.sh | 5 + .ci/scripts/wheel/pre_build_script.sh | 9 +- .ci/scripts/wheel/test_shared_libraries.py | 140 ++++++++++++++++- .github/workflows/build-presets.yml | 2 +- .github/workflows/build-wheels-macos.yml | 7 +- .github/workflows/mlx.yml | 52 ++++--- Makefile | 2 +- 8 files changed, 328 insertions(+), 39 deletions(-) diff --git a/.ci/scripts/tests/test_wheel_platform_tag.py b/.ci/scripts/tests/test_wheel_platform_tag.py index 415ef29582c..6b3db641003 100644 --- a/.ci/scripts/tests/test_wheel_platform_tag.py +++ b/.ci/scripts/tests/test_wheel_platform_tag.py @@ -4,28 +4,33 @@ # This source code is licensed under the BSD-style license found in the # LICENSE file in the root directory of this source tree. -"""Unit tests for the wheel platform tag comparison. +"""Unit tests for the wheel platform tag checks. -Here rather than in the wheel checks, because the decision under test is a pure -function of two strings. Running it as part of the wheel checks meant it needed eleven -built wheels to exercise one comparison, and it still could not run on a machine that -had not built one. +Here rather than in the wheel checks, because nothing under test needs a built wheel. +Running it as part of the wheel checks meant it needed eleven built wheels to exercise +one comparison, and it still could not run on a machine that had not built one. -The comparison had a real defect that this covers. The release pipeline builds in a +The tag comparison had a real defect that this covers. The release pipeline builds in a manylinux image and rewrites the wheel's file name, so the tag on the file and the tag auditwheel reports never agree in spelling, and comparing them as text rejected every correct wheel. No local build reproduces that rewrite, so nothing short of a unit test catches it before CI. """ +import subprocess import sys +import zipfile from pathlib import Path import pytest sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "wheel")) +import test_shared_libraries # noqa: E402 from test_shared_libraries import ( # noqa: E402 + _mach_o_minimum, + _macos_minimum_violations, + _metallib_minimum, _tag_architectures_match, _wheel_architecture, ) @@ -75,3 +80,136 @@ def test_reads_every_architecture_the_project_builds_for(): for architecture in ("x86_64", "aarch64", "i686", "ppc64le", "s390x", "armv7l"): assert _wheel_architecture(f"linux_{architecture}") == architecture assert _wheel_architecture(f"manylinux_2_28_{architecture}") == architecture + + +@pytest.mark.parametrize( + "tag,minimums,violations", + [ + # Binaries at or below the supported minimum. + ("macosx_14_0_arm64", {"a.dylib": (14, 0), "b.so": (13, 0)}, (False, [])), + # A newer major version, after bdist_wheel raised the tag to match it. + ("macosx_15_0_arm64", {"a.dylib": (15, 0)}, (True, ["a.dylib"])), + # A tag that promises an older macOS than the binaries support. + ("macosx_11_0_arm64", {"a.dylib": (14, 0)}, (False, ["a.dylib"])), + # A binary that declares no minimum cannot be shown to meet one. + ("macosx_14_0_arm64", {"a.dylib": None}, (False, ["a.dylib"])), + ], +) +def test_finds_what_stops_the_wheel_on_the_oldest_supported_macos( + tag, minimums, violations +): + assert _macos_minimum_violations(tag, minimums) == violations + + +# How otool -l prints the load command that holds a library's minimum macOS. +_BUILD_VERSION = """\ + cmd LC_BUILD_VERSION + cmdsize 32 + platform 1 + minos 14.0 + sdk 15.5 + ntools 1 + tool 3 + version 1167.5 +""" + +# The older command, which ld64 writes instead for targets below macOS 10.14. +_VERSION_MIN = """\ + cmd LC_VERSION_MIN_MACOSX + cmdsize 16 + version 10.13 + sdk 15.5 +""" + + +@pytest.mark.parametrize( + "load_commands,minimum", + [ + (_BUILD_VERSION, (14, 0)), + (_VERSION_MIN, (10, 13)), + # A file with neither command cannot show the minimum is met. + (" cmd LC_SEGMENT_64\n", None), + ], +) +def test_reads_the_minimum_macos_of_a_mach_o_file(load_commands, minimum): + assert _mach_o_minimum(load_commands) == minimum + + +@pytest.mark.parametrize( + "data,minimum", + [ + # Newer compilers add a version to the architecture. The newest target counts. + (b"air64_v26-apple-macosx14.0.0\0air64_v26-apple-macosx15.0.0\0", (15, 0)), + # A Metal library that names no target cannot show the minimum is met. + (b"MTLB" + b"\0" * 64, None), + ], +) +def test_reads_the_minimum_macos_of_a_metal_library(data, minimum): + assert _metallib_minimum(data) == minimum + + +# Fake wheel contents by path. A Mach-O file is its header followed by the minimum +# macOS of each architecture, which the fake otool prints back. +_THIN, _FAT64 = b"\xcf\xfa\xed\xfe", b"\xca\xfe\xba\xbf" +_WHEEL = { + "executorch/lib/libexecutorch.dylib": _THIN + b"14.0", + "executorch/data/bin/flatc": _FAT64 + b"14.0 14.0", + "executorch/lib/mlx.metallib": b"MTLB air64-apple-macosx14.0.0", +} + + +@pytest.mark.parametrize( + "tag,files,error", + [ + ("macosx_14_0_arm64", _WHEEL, None), + # The second architecture of a universal executable needs macOS 14.5. + ( + "macosx_14_0_arm64", + {**_WHEEL, "executorch/data/bin/flatc": _FAT64 + b"14.0 14.5"}, + r"flatc \(14\.5\)", + ), + ( + "macosx_14_0_arm64", + {**_WHEEL, "executorch/lib/mlx.metallib": b"MTLB air64-apple-macosx14.5.0"}, + r"mlx\.metallib \(14\.5\)", + ), + ("macosx_15_0_arm64", _WHEEL, "tagged for a macOS newer than 14.0"), + ( + "macosx_14_0_arm64", + {**_WHEEL, "executorch/lib/libexecutorch.dylib": _THIN}, + "otool could not read", + ), + ( + "macosx_14_0_arm64", + {"executorch/lib/libexecutorch.dylib": b"not a Mach-O file"}, + "contains no Mach-O files or Metal libraries", + ), + ], +) +def test_checks_every_binary_in_a_macos_wheel(tmp_path, monkeypatch, tag, files, error): + # The check runs on macOS after a wheel build, but all it reads is a zip file and + # what lipo and otool print, so here the wheel and both tools are fake. + wheel = tmp_path / f"executorch-1.0-cp311-cp311-{tag}.whl" + with zipfile.ZipFile(wheel, "w") as archive: + for name, data in files.items(): + archive.writestr(name, data) + + def run(command, **kwargs): + if command[0] == "lipo": + return subprocess.CompletedProcess(command, 0, "arm64\n", "") + minimums = Path(command[-1]).read_bytes()[4:].decode().split() + # Without -arch all, otool can print just one architecture of a universal file. + if "all" not in command: + minimums = minimums[:1] + printed = "".join(f"cmd LC_BUILD_VERSION\nminos {m}\n" for m in minimums) + return subprocess.CompletedProcess(command, 0 if minimums else 1, printed, "") + + monkeypatch.setattr(sys, "platform", "darwin") + monkeypatch.setattr(test_shared_libraries, "_find_wheel_files", lambda: [wheel]) + monkeypatch.setattr(test_shared_libraries, "_tool", lambda name: name) + monkeypatch.setattr(subprocess, "run", run) + if error is None: + test_shared_libraries.test_wheel_platform_tag() + else: + with pytest.raises(AssertionError, match=error): + test_shared_libraries.test_wheel_platform_tag() diff --git a/.ci/scripts/wheel/envvar_macos.sh b/.ci/scripts/wheel/envvar_macos.sh index f72bcd2d46c..97bea1dacc7 100755 --- a/.ci/scripts/wheel/envvar_macos.sh +++ b/.ci/scripts/wheel/envvar_macos.sh @@ -14,3 +14,8 @@ source "${GITHUB_WORKSPACE}/${REPOSITORY}/.ci/scripts/wheel/envvar_base.sh" # Apple SDK Objective-C framework headers (e.g. NSIntegerMax in NSObjCRuntime.h). export CC=/usr/bin/clang export CXX=/usr/bin/clang++ + +# Pin Xcode so a runner image update cannot change the wheel unnoticed. Its SDK +# decides which Core ML code is built in. 16.4 is the macOS 15 image's default. The +# 26.0.1 pins in other workflows build PyTorch from source, which this build does not. +export DEVELOPER_DIR=/Applications/Xcode_16.4.app/Contents/Developer diff --git a/.ci/scripts/wheel/pre_build_script.sh b/.ci/scripts/wheel/pre_build_script.sh index fcbcd2069ff..1ad8ee8e23f 100755 --- a/.ci/scripts/wheel/pre_build_script.sh +++ b/.ci/scripts/wheel/pre_build_script.sh @@ -179,15 +179,18 @@ if [[ $UNAME_S == *"MINGW"* || $UNAME_S == *"MSYS"* ]]; then fi fi -# Enable VGF in pybind wheel builds when the platform-specific build input is -# available from pip. +# Enable VGF in Linux pybind wheel builds when the platform-specific build input +# is available from pip. Not on macOS: the VGF library's macOS wheels need macOS 15, +# newer than the oldest macOS this wheel supports. if [[ "$UNAME_S" == "Linux" || "$UNAME_S" == "Darwin" ]]; then # The wheel rewrites recorded runtime paths after linking, which needs patchelf. It is declared # under build-system requires, but that is honoured only by a frontend with build isolation and # this path deliberately installs requirements itself, so without this the rewrite silently # does nothing and a CUDA wheel loses its route to the NVIDIA libraries beside it. python3 -m pip install patchelf || true - if python3 -m pip install -r \ + if [[ "$UNAME_S" == "Darwin" ]]; then + echo "VGF needs a newer macOS than this wheel supports; building without VGF" + elif python3 -m pip install -r \ "${GITHUB_WORKSPACE}/${REPOSITORY}/backends/arm/requirements-arm-vgf-runtime.txt"; then export EXECUTORCH_PYBIND_ENABLE_VGF=ON echo "EXECUTORCH_PYBIND_ENABLE_VGF=ON" >> "${GITHUB_ENV}" diff --git a/.ci/scripts/wheel/test_shared_libraries.py b/.ci/scripts/wheel/test_shared_libraries.py index 55120809626..b9ad0149ec1 100644 --- a/.ci/scripts/wheel/test_shared_libraries.py +++ b/.ci/scripts/wheel/test_shared_libraries.py @@ -194,6 +194,8 @@ b"\xfe\xed\xfa\xce", b"\xca\xfe\xba\xbe", b"\xbe\xba\xfe\xca", + b"\xca\xfe\xba\xbf", + b"\xbf\xba\xfe\xca", } ) @@ -769,6 +771,127 @@ def _assert_mach_o_architecture_matches(wheel: Path) -> None: print(f"\u2713 the wheel is tagged for the architecture it contains ({declared})") +# The oldest macOS the wheel supports. The pybind preset sets it, and so does the +# MACOSX_DEPLOYMENT_TARGET that the pytorch/test-infra wheel build exports. +_MACOS_MINIMUM = (14, 0) + + +def _macos_minimum_violations(tag: str, minimums: dict) -> tuple[bool, list]: + """Whether a macOS tag asks for more than the supported minimum, and which binaries + need more than the tag or the supported minimum allows. + + Returned rather than asserted so the decision can be unit tested without a wheel. + `minimums` maps each binary to the minimum macOS it declares, or to None when it + declares none, which counts against it. + """ + # In macosx_14_0_arm64 the two numbers after the platform name are the version. + tagged = tuple(int(part) for part in tag.split("_")[1:3]) + allowed = min(tagged, _MACOS_MINIMUM) + return tagged > _MACOS_MINIMUM, sorted( + name + for name, minimum in minimums.items() + if minimum is None or minimum > allowed + ) + + +def _metallib_minimum(data: bytes) -> tuple[int, int] | None: + """The newest macOS that a kernel in a Metal library targets, or None if none says. + + Each kernel keeps its target, such as air64-apple-macosx14.0.0, or + air64_v26-apple-macosx14.0.0 from newer compilers. + """ + versions = re.findall(rb"air64[\w.]*-apple-macosx(\d+)\.(\d+)", data) + return max(((int(major), int(minor)) for major, minor in versions), default=None) + + +def _mach_o_minimum(load_commands: str) -> tuple[int, int] | None: + """The newest minimum macOS in `otool -l` output, or None if it declares none. + + A universal binary declares one for each architecture. Targets below macOS 10.14 + store it as the version of the older LC_VERSION_MIN_MACOSX command. + """ + versions = re.findall( + r"(?:LC_BUILD_VERSION.*?minos|LC_VERSION_MIN_MACOSX.*?version) (\d+)\.(\d+)", + load_commands, + re.DOTALL, + ) + return max(((int(major), int(minor)) for major, minor in versions), default=None) + + +def _assert_minimum_macos(wheel: Path) -> None: + """Fail if a macOS wheel needs a newer macOS than the project supports. + + bdist_wheel raises the tag, and only warns, when a .so or .dylib needs a newer major + macOS, so pip on an older macOS quietly picks an older release. It ignores a newer + minor version, executables such as flatc, and Metal libraries, so those install and + then fail to load. Installing the wheel on the oldest supported macOS used to catch + both, and no build runner runs that macOS any more. This reads the minimum that each + Mach-O file and Metal library declares, so it cannot see a call to a newer macOS API + that skips its availability check, or code built for a newer macOS that was linked + into a library, since the linker records the library's own minimum and only warns. + """ + otool = _tool("otool") + assert otool is not None, ( + "otool is required to read the minimum macOS of the wheel's binaries, and it " + "was not found" + ) + claimed = wheel.name.split("-")[-1].removesuffix(".whl") + floor = ".".join(str(part) for part in _MACOS_MINIMUM) + + minimums, unreadable = {}, [] + with tempfile.TemporaryDirectory() as unpacked: + with zipfile.ZipFile(wheel) as archive: + archive.extractall(unpacked) + root = Path(unpacked) + for path in sorted(root.rglob("*")): + if not path.is_file() or path.is_symlink(): + continue + name = str(path.relative_to(root)) + if path.suffix == ".metallib": + minimum = _metallib_minimum(path.read_bytes()) + else: + with path.open("rb") as handle: + if handle.read(4) not in _MACH_O_MAGIC: + continue + result = subprocess.run( + [otool, "-arch", "all", "-l", str(path)], + capture_output=True, + text=True, + check=False, + ) + if result.returncode != 0: + unreadable.append(f"{name}: {result.stderr.strip()[:120]}") + continue + minimum = _mach_o_minimum(result.stdout) + minimums[name] = minimum + + assert not unreadable, ( + f"otool could not read these Mach-O files in {wheel.name}, so the minimum macOS " + f"check covered less than the wheel ships: {unreadable}" + ) + assert minimums, ( + f"the wheel {wheel.name} contains no Mach-O files or Metal libraries, so " + f"nothing was checked against macOS {floor}" + ) + tag_too_new, too_new = _macos_minimum_violations(claimed, minimums) + assert not too_new, ( + f"these files in {wheel.name} declare no minimum macOS, or a newer one than " + f"its tag or macOS {floor} allows: " + + ", ".join( + f"{name} ({'.'.join(map(str, minimums[name] or ())) or 'none'})" + for name in too_new + ) + ) + assert not tag_too_new, ( + f"the wheel {wheel.name} is tagged for a macOS newer than {floor}, so pip on " + f"macOS {floor} will not install it" + ) + print( + f"\u2713 the wheel's tag allows macOS {floor} and no Mach-O file or Metal " + "library in it declares a newer minimum" + ) + + def _shipped_object_patterns() -> list[str]: """Filename patterns for every shipped loadable object. @@ -2324,16 +2447,20 @@ def _tag_architectures_match(claimed: str, supported: str): def test_wheel_platform_tag() -> None: """The wheel's declared platform tag must name the architecture it was built for. - Only the architecture. auditwheel cannot certify a glibc baseline for this wheel: - it depends on torch without vendoring torch's libraries, so the contents reference - libtorch.so from outside any manylinux policy and auditwheel reports a plain - linux_. That is the expected answer for a torch-dependent wheel rather than - a defect, and the manylinux tag on the file comes from the build image. Asserting - the baseline here failed every correct wheel. + On Linux, only the architecture. auditwheel cannot certify a glibc baseline for + this wheel: it depends on torch without vendoring torch's libraries, so the contents + reference libtorch.so from outside any manylinux policy and auditwheel reports a + plain linux_. That is the expected answer for a torch-dependent wheel rather + than a defect, and the manylinux tag on the file comes from the build image. + Asserting the baseline here failed every correct wheel. The architecture is still worth checking, because a wheel labelled with the wrong one installs on machines it cannot run on at all, and that is a mistake this can actually catch. + + On macOS the tag also names the oldest macOS the wheel installs on. That is + checked too, because a build that moves it only warns, and no CI job runs that + macOS any more. """ wheels = _find_wheel_files() if not wheels: @@ -2353,6 +2480,7 @@ def test_wheel_platform_tag() -> None: # correct wheel to fail the job. if sys.platform == "darwin": _assert_mach_o_architecture_matches(wheels[-1]) + _assert_minimum_macos(wheels[-1]) return if _WINDOWS: _assert_pe_architecture_matches(wheels[-1]) diff --git a/.github/workflows/build-presets.yml b/.github/workflows/build-presets.yml index abdc8647ac7..e8714716eb4 100644 --- a/.github/workflows/build-presets.yml +++ b/.github/workflows/build-presets.yml @@ -27,7 +27,7 @@ jobs: default-packages: "" job-name: build ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} - runner: macos-14-xlarge + runner: macos-15-xlarge python-version: 3.12 submodules: recursive timeout: 90 diff --git a/.github/workflows/build-wheels-macos.yml b/.github/workflows/build-wheels-macos.yml index b632f7afdee..5677452a21a 100644 --- a/.github/workflows/build-wheels-macos.yml +++ b/.github/workflows/build-wheels-macos.yml @@ -7,6 +7,11 @@ on: - .ci/**/* - .github/workflows/build-wheels-macos.yml - '**/CMakeLists.txt' + # The pybind preset sets the oldest macOS the wheel supports, and the MLX patches + # and submodule set the one its Metal library declares. + - CMakePresets.json + - backends/mlx/patches/** + - backends/mlx/third-party/mlx - examples/**/* - pyproject.toml - setup.py @@ -74,7 +79,7 @@ jobs: post-script: ${{ matrix.post-script }} package-name: ${{ matrix.package-name }} # Meta's macOS runners do not have Xcode, so use GitHub's runners. - runner-type: macos-14-xlarge + runner-type: macos-15-xlarge setup-miniconda: true smoke-test-script: ${{ matrix.smoke-test-script }} trigger-event: ${{ github.event_name }} diff --git a/.github/workflows/mlx.yml b/.github/workflows/mlx.yml index f01fd3d3cbc..566677d489e 100644 --- a/.github/workflows/mlx.yml +++ b/.github/workflows/mlx.yml @@ -54,7 +54,7 @@ jobs: with: default-packages: "" job-name: test-mlx - runner: macos-14-xlarge + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} @@ -159,13 +159,15 @@ jobs: with: default-packages: "" job-name: test-mlx-supertonic - runner: macos-14-xlarge + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} timeout: 90 script: | set -eux + # Top-level presets build with unlimited make -j and per-model presets serially. + export CMAKE_BUILD_PARALLEL_LEVEL=$(( $(sysctl -n hw.ncpu) - 1 )) echo "::group::Install ExecuTorch and Supertonic requirements" ${CONDA_RUN} python install_executorch.py > /dev/null @@ -246,13 +248,15 @@ jobs: with: default-packages: "" job-name: test-mlx-qwen35-moe - runner: macos-14-xlarge + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} timeout: 90 script: | set -eux + # Top-level presets build with unlimited make -j and per-model presets serially. + export CMAKE_BUILD_PARALLEL_LEVEL=$(( $(sysctl -n hw.ncpu) - 1 )) echo "::group::Install ExecuTorch" ${CONDA_RUN} python install_executorch.py > /dev/null @@ -327,13 +331,15 @@ jobs: with: default-packages: "" job-name: test-mlx-muse-glimmer-build - runner: macos-14-xlarge + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} timeout: 90 script: | set -eux + # Top-level presets build with unlimited make -j and per-model presets serially. + export CMAKE_BUILD_PARALLEL_LEVEL=$(( $(sysctl -n hw.ncpu) - 1 )) echo "::group::Install ExecuTorch" ${CONDA_RUN} python install_executorch.py > /dev/null @@ -366,7 +372,7 @@ jobs: BATCHING_BIN_DIR="${PWD}/cmake-out/examples/models/muse-glimmer-batching" ${CONDA_RUN} cmake -S "${PWD}/examples/models/muse-glimmer" -B "${BATCHING_BIN_DIR}" \ -DMUSE_GLIMMER_BUILD_BATCHING_TESTS=ON - ${CONDA_RUN} cmake --build "${BATCHING_BIN_DIR}" --target muse_glimmer_batching_test --parallel + ${CONDA_RUN} cmake --build "${BATCHING_BIN_DIR}" --target muse_glimmer_batching_test ${CONDA_RUN} ctest --test-dir "${BATCHING_BIN_DIR}" \ -R '^muse_glimmer_batching$' --output-on-failure --no-tests=error test -x "${BATCHING_BIN_DIR}/muse_glimmer_batching_runner" @@ -383,13 +389,15 @@ jobs: with: default-packages: "" job-name: test-mlx-nemotron3-diarization-build - runner: macos-14-xlarge + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} timeout: 90 script: | set -eux + # Top-level presets build with unlimited make -j and per-model presets serially. + export CMAKE_BUILD_PARALLEL_LEVEL=$(( $(sysctl -n hw.ncpu) - 1 )) ${CONDA_RUN} python install_executorch.py > /dev/null ${CONDA_RUN} make nemotron3-diarization-mlx BIN_DIR=cmake-out/examples/models/nemotron3_diarization @@ -412,7 +420,7 @@ jobs: with: default-packages: "" job-name: test-mlx-backend-${{ matrix.suite }} - runner: macos-14-xlarge + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} @@ -458,13 +466,15 @@ jobs: with: default-packages: "" job-name: test-mlx-parakeet - runner: macos-14-xlarge + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} timeout: 90 script: | set -eux + # Top-level presets build with unlimited make -j and per-model presets serially. + export CMAKE_BUILD_PARALLEL_LEVEL=$(( $(sysctl -n hw.ncpu) - 1 )) echo "::group::Install ExecuTorch" ${CONDA_RUN} python install_executorch.py > /dev/null @@ -520,7 +530,7 @@ jobs: with: default-packages: "" job-name: test-mlx-voxtral - runner: macos-14-xlarge + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} @@ -528,6 +538,8 @@ jobs: timeout: 90 script: | set -eux + # Top-level presets build with unlimited make -j and per-model presets serially. + export CMAKE_BUILD_PARALLEL_LEVEL=$(( $(sysctl -n hw.ncpu) - 1 )) echo "::group::Install ExecuTorch" ${CONDA_RUN} python install_executorch.py > /dev/null @@ -584,7 +596,7 @@ jobs: with: default-packages: "" job-name: test-mlx-voxtral-realtime - runner: macos-14-xlarge + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} @@ -592,6 +604,8 @@ jobs: timeout: 90 script: | set -eux + # Top-level presets build with unlimited make -j and per-model presets serially. + export CMAKE_BUILD_PARALLEL_LEVEL=$(( $(sysctl -n hw.ncpu) - 1 )) # Disable HF Xet storage to avoid stalled downloads on CI runners export HF_HUB_DISABLE_XET=1 @@ -665,7 +679,7 @@ jobs: with: default-packages: "" job-name: test-mlx-whisper - runner: macos-14-xlarge + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} @@ -719,13 +733,15 @@ jobs: with: default-packages: "" job-name: test-mlx-stories110m - runner: macos-14-xlarge + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} timeout: 90 script: | set -eux + # Top-level presets build with unlimited make -j and per-model presets serially. + export CMAKE_BUILD_PARALLEL_LEVEL=$(( $(sysctl -n hw.ncpu) - 1 )) echo "::group::Install ExecuTorch" ${CONDA_RUN} python install_executorch.py > /dev/null @@ -797,26 +813,23 @@ jobs: name: "gemma3-1b" use-custom: [false, true] qconfig: ["4w", "nvfp4"] - runner: ["macos-14-xlarge"] include: - model: id: "google/gemma-4-E2B-it" name: "gemma4-e2b" use-custom: true qconfig: "4w" - runner: "macos-15-xlarge" - model: id: "google/gemma-4-E2B-it" name: "gemma4-e2b" use-custom: false qconfig: "4w" - runner: "macos-15-xlarge" uses: pytorch/test-infra/.github/workflows/macos_job.yml@main secrets: inherit with: default-packages: "" job-name: test-mlx-llm-${{ matrix.model.name }}${{ matrix.use-custom && '-custom' || '' }}-${{ matrix.qconfig }} - runner: ${{ matrix.runner }} + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} @@ -900,7 +913,7 @@ jobs: with: default-packages: "" job-name: test-mlx-dflash - runner: macos-14-xlarge + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} @@ -1027,23 +1040,20 @@ jobs: - id: "unsloth/Llama-3.2-1B-Instruct" name: "llama-1b" chat: "llama3" - runner: "macos-14-xlarge" - id: "unsloth/gemma-3-1b-it" name: "gemma3-1b" chat: "gemma" - runner: "macos-14-xlarge" # Only model with KV sharing: 15 caches for 35 layers, and sliding and # full-attention caches with different head shapes. - id: "google/gemma-4-E2B-it" name: "gemma4-e2b" chat: "gemma4" - runner: "macos-15-xlarge" uses: pytorch/test-infra/.github/workflows/macos_job.yml@main secrets: inherit with: default-packages: "" job-name: test-mlx-llm-offgraph-${{ matrix.model.name }} - runner: ${{ matrix.model.runner }} + runner: macos-15-xlarge python-version: "3.12" submodules: recursive ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} diff --git a/Makefile b/Makefile index dfac9ff09ac..a021214ab7b 100644 --- a/Makefile +++ b/Makefile @@ -539,7 +539,7 @@ muse-glimmer-batching-mlx: -DMUSE_GLIMMER_BUILD_BATCHING_TESTS=OFF cmake --build "$(CURDIR)/cmake-out/examples/models/muse-glimmer-batching" \ --target muse_glimmer_batching_runner muse_glimmer_batching_worker \ - --parallel "$$(sysctl -n hw.ncpu)" + --parallel "$${CMAKE_BUILD_PARALLEL_LEVEL:-$$(sysctl -n hw.ncpu)}" @echo "" @echo "Build complete!" @echo " Runner: $(CURDIR)/cmake-out/examples/models/muse-glimmer-batching/muse_glimmer_batching_runner"