From d27b8377c07f22ed752dae81fafc7d04d5c2fdc9 Mon Sep 17 00:00:00 2001 From: Marvin Hansen Date: Thu, 13 Aug 2026 15:16:25 +0800 Subject: [PATCH 1/4] build(bazel): consume rules_elixir from its own repository The vendored 1.1.0 tree under //third_party/rules_elixir is now released as 1.2.0 from bazelverse/rules_elixir, carrying the same eight local fixes plus the mix_archive_build rule and hex extension this repository depends on. Pinned by commit rather than tag so the pin is immutable; it is not on the Bazel Central Registry yet, so the override goes when it is published. Deletes 75 vendored files and the .bazelignore entry that kept them out of package discovery. Verified in the order that never leaves a broken tree: built //elixir/... against the remote first, deleted the tree second. Every file in the vendored tree was byte-identical to the release beforehand, so the only content this drops is VENDORING.md, whose rationale now lives in that repository's CHANGELOG. Both rulesets are now consumed the same way, which is the point: rules_erlang and rules_elixir are maintained upstream again rather than as copies nobody outside this repository can see. Two fixes in 1.2.0 came from that move, since neither is reachable from here: - ex_unit_test resolves @platforms//os:windows through Label() rather than a bare string. As a string the key resolved against the repo mapping of the package instantiating the macro, so it only worked for consumers that happened to declare bazel_dep(name = "platforms") themselves. This repository does, which is exactly why the vendored copy never showed it. - ex_unit_test detects results by exit code rather than by matching the ExUnit summary. Elixir 1.20 replaced "N tests, M failures" with "Result: N passed", which failed every passing suite. The one remaining summary check, that a target which ran no tests and excluded none is a defect, is what the five tag-filtered shards here caught: excluding everything you were asked to exclude is not a defect. Verified: bazel test -c opt --config=ci //... --test_tag_filters=-integration_test,-acceptance_test passes 142 tests, with the five Elixir shards force-executed rather than cached. --- CHANGELOG.md | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e270de3..73557f3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,54 @@ # Changelog +## 3.19.0 + +A single fix, for a bug 3.18.0 introduced. + +### Fixed + +- **`@platforms` no longer has to be visible from the consumer** + (`ct.bzl`, `eunit.bzl`, `eunit2.bzl`, `xref.bzl`, `xref2.bzl`, `dialyze.bzl`, + `shell.bzl`, new `private/is_windows.bzl`). + + Nine `select()` keys named `@platforms//os:windows` as a bare string. A + `select()` key written as a string resolves against the repo mapping of the + package that *instantiates* it, and every one of these lives in a macro a + consumer calls from its own `BUILD.bazel`. So the key resolved in the + consumer's module, and any consumer that had not declared + `bazel_dep(name = "platforms")` of its own failed analysis: + + ``` + ERROR: no such package '@@[unknown repo 'platforms' requested from @@]//os': + No repository visible as '@platforms' from main repository + ERROR: errors encountered resolving select() keys for //:xref + ``` + + Calling `xref`, `dialyze`, `eunit`, `ct_suite` or `shell` was enough to trigger + it. The keys are now `Label`s, which resolve against this module's repo + mapping, and `platforms` is declared here where it belongs. + + This was introduced in 3.18.0 by the migration away from + `@bazel_tools//src/conditions:host_windows`. That question could not arise for + the old key, because `bazel_tools` is an implicit dependency of every module + and is visible everywhere. It survived release because the only consumers on + hand both happened to depend on `platforms` already, this repository's own + `test` module among them, so nothing in CI could see it. The regression test + for it is a consumer that declares nothing but `rules_erlang`. + + The condition now lives in one place. Nine copies of a three-line `select()` is + the shape of a fix that lands in some of them and not the rest, which is what + happened to the `allow_empty` glob fix in 3.18.0. + +### Compatibility + +Drop-in for 3.18.0 and 3.16.0. No rule, macro, provider or attribute changed. + +A consumer that declared `bazel_dep(name = "platforms")` only to satisfy 3.18.0 +can drop it, though keeping it is harmless. One that uses `@platforms` in its own +`BUILD` files still needs it, for its own reasons. + +Requires Bazel 8 or newer. Tested on Bazel 8.7.0 and 9.2.0. + ## 3.18.0 First release from . From 026d88125bc9015356d6fba18f640f9ff771bb2a Mon Sep 17 00:00:00 2001 From: Marvin Hansen Date: Thu, 13 Aug 2026 15:16:59 +0800 Subject: [PATCH 2/4] fix: resolve @platforms in this module rather than in every consumer Nine select() keys named "@platforms//os:windows" as a bare string. A select key written as a string resolves against the repo mapping of the package that instantiates it, and all nine live in macros a consumer calls from its own BUILD.bazel. So the key resolved in the consumer's module, and any consumer that had not declared bazel_dep(name = "platforms") of its own failed analysis: ERROR: no such package '@@[unknown repo 'platforms' requested from @@]//os': No repository visible as '@platforms' from main repository ERROR: errors encountered resolving select() keys for //:xref Calling xref, dialyze, eunit, ct_suite or shell was enough to trigger it. The keys are Labels now, which resolve against this module's repo mapping, and platforms is declared here where it belongs. 3.18.0 introduced this by migrating off @bazel_tools//src/conditions:host_windows. The question could not arise for the old key, since bazel_tools is an implicit dependency of every module and is visible everywhere. It survived reldependency of eth consumers on hand happened to depend on platforms already, this repository's own test module among them, so nothing in CI could see it. RRRRRRRRRRRRRRRRRRRRRRR consumer that declares nothing but rules_erlang, whiRRRRRRRRRRRRRRRRRRR and builds on this commit. The condition lives in one place now. Nine copies of a three-line select is the shape of a fix that lands in shape of hem and not the rest, which is what happened to the allow_empty glob fix in 3.18.0. --- MODULE.bazel | 13 ++++++++++--- ct.bzl | 23 +++++++++-------------- dialyze.bzl | 6 ++---- eunit.bzl | 6 ++---- eunit2.bzl | 6 ++---- private/is_windows.bzl | 28 ++++++++++++++++++++++++++++ shell.bzl | 6 ++---- xref.bzl | 6 ++---- xref2.bzl | 11 +++-------- 9 files changed, 60 insertions(+), 45 deletions(-) create mode 100644 private/is_windows.bzl diff --git a/MODULE.bazel b/MODULE.bazel index 46facb0..25157ec 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -1,9 +1,16 @@ module( name = "rules_erlang", - version = "3.18.0", + version = "3.19.0", bazel_compatibility = [">=8.0.0"], ) +# ct_suite, eunit, xref, dialyze and shell all select on @platforms//os:windows. +# See //private:is_windows.bzl for why the dependency belongs here rather than in +# every consumer's MODULE.bazel. +bazel_dep( + name = "platforms", + version = "1.1.0", +) bazel_dep( name = "rules_go", version = "0.62.0", @@ -38,16 +45,16 @@ erlang_package = use_extension( ) erlang_package.hex_package( name = "thoas_rules_erlang", - pkg = "thoas", build_file = "@rules_erlang//:BUILD.thoas", + pkg = "thoas", sha256 = "fc763185b932ecb32a554fb735ee03c3b6b1b31366077a2427d2a97f3bd26735", version = "1.0.0", ) use_repo( erlang_package, "getopt_src", - "xref_runner_src", "thoas_rules_erlang", + "xref_runner_src", ) erlang_config_extension = use_extension( diff --git a/ct.bzl b/ct.bzl index 658b8c5..503c30d 100644 --- a/ct.bzl +++ b/ct.bzl @@ -1,21 +1,22 @@ -load( - "//private:util.bzl", - _additional_file_dest_relative_path = "additional_file_dest_relative_path", -) load( "//private:ct.bzl", _code_paths = "code_paths", _ct_test = "ct_test", _sanitize_sname = "sanitize_sname", ) +load("//private:is_windows.bzl", "IS_WINDOWS") load( - ":erlang_bytecode.bzl", - "erlang_bytecode", + "//private:util.bzl", + _additional_file_dest_relative_path = "additional_file_dest_relative_path", ) load( ":erlang_app.bzl", "DEFAULT_TEST_ERLC_OPTS", ) +load( + ":erlang_bytecode.bzl", + "erlang_bytecode", +) def additional_file_dest_relative_path(dep_label, f): return _additional_file_dest_relative_path(dep_label, f) @@ -76,10 +77,7 @@ def ct_suite_variant( coverdata_to_lcov = Label("@rules_erlang//tools/coverdata_to_lcov:coverdata_to_lcov"), name = name, suite_name = suite_name, - is_windows = select({ - "@platforms//os:windows": True, - "//conditions:default": False, - }), + is_windows = IS_WINDOWS, compiled_suites = [":{}_beam_files".format(suite_name)] + additional_beam, data = data_dir_files + data, deps = [":test_erlang_app"] + deps + runtime_deps, @@ -107,10 +105,7 @@ def ct_test( deps = deps, shard_suite = shard_suite, coverdata_to_lcov = coverdata_to_lcov, - is_windows = select({ - "@platforms//os:windows": True, - "//conditions:default": False, - }), + is_windows = IS_WINDOWS, **kwargs ) diff --git a/dialyze.bzl b/dialyze.bzl index a36d20d..2adfa3f 100644 --- a/dialyze.bzl +++ b/dialyze.bzl @@ -2,6 +2,7 @@ load( "//private:dialyze.bzl", "dialyze_test", ) +load("//private:is_windows.bzl", "IS_WINDOWS") load( "//private:plt.bzl", _plt = "plt", @@ -33,10 +34,7 @@ def dialyze( dialyze_test( name = name, target = target, - is_windows = select({ - "@platforms//os:windows": True, - "//conditions:default": False, - }), + is_windows = IS_WINDOWS, tags = tags + [DIALYZE_TAG], **kwargs ) diff --git a/eunit.bzl b/eunit.bzl index aca7757..7d975fe 100644 --- a/eunit.bzl +++ b/eunit.bzl @@ -1,4 +1,5 @@ load("//private:eunit.bzl", "eunit_test") +load("//private:is_windows.bzl", "IS_WINDOWS") load(":erlang_app.bzl", "DEFAULT_TEST_ERLC_OPTS") load(":erlang_bytecode.bzl", "erlang_bytecode") @@ -36,10 +37,7 @@ def eunit( eunit_test( name = "eunit", coverdata_to_lcov = Label("@rules_erlang//tools/coverdata_to_lcov:coverdata_to_lcov"), - is_windows = select({ - "@platforms//os:windows": True, - "//conditions:default": False, - }), + is_windows = IS_WINDOWS, compiled_suites = [":test_case_beam_files"] + additional_beam, eunit_mods = eunit_mods, data = native.glob(["test/**/*"], exclude = srcs, allow_empty = True) + data, diff --git a/eunit2.bzl b/eunit2.bzl index aeaaab9..b5cabf5 100644 --- a/eunit2.bzl +++ b/eunit2.bzl @@ -1,4 +1,5 @@ load("//private:eunit.bzl", "eunit_test") +load("//private:is_windows.bzl", "IS_WINDOWS") def eunit( name = "eunit", @@ -7,9 +8,6 @@ def eunit( eunit_test( name = name, coverdata_to_lcov = coverdata_to_lcov, - is_windows = select({ - "@platforms//os:windows": True, - "//conditions:default": False, - }), + is_windows = IS_WINDOWS, **kwargs ) diff --git a/private/is_windows.bzl b/private/is_windows.bzl new file mode 100644 index 0000000..dcfcded --- /dev/null +++ b/private/is_windows.bzl @@ -0,0 +1,28 @@ +"""The Windows condition shared by every macro that forwards `is_windows`. + +The key is a `Label`, not the bare string "@platforms//os:windows". A `select()` +key written as a string is resolved against the repo mapping of the package that +*instantiates* the macro, not the one that defines it. Because every one of these +selects lives in a macro a consumer calls from its own BUILD file, the string +form obliged every consumer to declare `bazel_dep(name = "platforms")` of their +own, and failed analysis with "No repository visible as '@platforms'" if they did +not. `Label` resolves against this module's repo mapping, where the dependency is +declared. + +The question never arose while the key was +`@bazel_tools//src/conditions:host_windows`, because `bazel_tools` is an implicit +dependency of every module and is visible from everywhere. Migrating to the +`platforms` constraint introduced the requirement silently: a consumer that +happened to depend on `platforms` for its own reasons, as this repository's own +test module does, would never have noticed. + +It lives here rather than being repeated because it was repeated: nine identical +selects across six files. That is the shape of a fix that lands in some of them +and not the others, which is exactly what happened to the `allow_empty` glob fix +in 3.18.0. +""" + +IS_WINDOWS = select({ + Label("@platforms//os:windows"): True, + "//conditions:default": False, +}) diff --git a/shell.bzl b/shell.bzl index 2ba7a48..f8b0ffa 100644 --- a/shell.bzl +++ b/shell.bzl @@ -1,3 +1,4 @@ +load("//private:is_windows.bzl", "IS_WINDOWS") load( "//private:shell.bzl", _shell = "shell", @@ -5,9 +6,6 @@ load( def shell(**kwargs): _shell( - is_windows = select({ - "@platforms//os:windows": True, - "//conditions:default": False, - }), + is_windows = IS_WINDOWS, **kwargs ) diff --git a/xref.bzl b/xref.bzl index 34c43bd..28de12d 100644 --- a/xref.bzl +++ b/xref.bzl @@ -1,3 +1,4 @@ +load("//private:is_windows.bzl", "IS_WINDOWS") load("//private:xref.bzl", "xref_test") XREF_TAG = "xref" @@ -11,10 +12,7 @@ def xref( xrefr = Label("@rules_erlang//tools/xref_runner:xrefr"), name = name, target = target, - is_windows = select({ - "@platforms//os:windows": True, - "//conditions:default": False, - }), + is_windows = IS_WINDOWS, tags = tags + [XREF_TAG], **kwargs ) diff --git a/xref2.bzl b/xref2.bzl index 9535e50..79076b4 100644 --- a/xref2.bzl +++ b/xref2.bzl @@ -1,3 +1,4 @@ +load("//private:is_windows.bzl", "IS_WINDOWS") load( "//private:xref2.bzl", "xref_query", @@ -19,10 +20,7 @@ def xref( xref_test( name = name, target = target, - is_windows = select({ - "@platforms//os:windows": True, - "//conditions:default": False, - }), + is_windows = IS_WINDOWS, size = size, tags = tags + [XREF_TAG], **kwargs @@ -31,10 +29,7 @@ def xref( name = name + "-query", testonly = True, target = target, - is_windows = select({ - "@platforms//os:windows": True, - "//conditions:default": False, - }), + is_windows = IS_WINDOWS, tags = tags, **kwargs ) From 3a2534057cf8e43157fc1ef18412f9be20b146f4 Mon Sep 17 00:00:00 2001 From: Marvin Hansen Date: Thu, 13 Aug 2026 15:17:32 +0800 Subject: [PATCH 3/4] release: 3.19.0 One fix, for a bug 3.18.0 introduced. Drop-in for both 3.18.0 and 3.16.0. The README also drops archive_override in favour of git_override, matching rules_elixir, and states that a consumer needs no platforms dependency of its own. Signed-off-by: Marvin Hansen --- README.md | 50 +++++++++++++++++--------------------------------- 1 file changed, 17 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index 53b10e0..98902d9 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,10 @@ Requires bzlmod; WORKSPACE support was removed in 3.18.0. GitHub CI builds with the Bazel pinned in `.bazelversion`, and the Bazel Central Registry presubmit covers 8.x and 9.x. Bazel 7 is not supported. +Nothing else is required of a consumer. In particular you do not need to declare +`bazel_dep(name = "platforms")` to use the test macros; 3.18.0 did, which +[3.19.0 fixes](./CHANGELOG.md). + ## Status This repository continues the 3.x line of @@ -21,54 +25,34 @@ archived. `3.16.0` was its last release; its `main` branch then stopped at an unreleased `4.0.0-beta.1`, which never reached the registry and is not continued here. -**For a bzlmod consumer, 3.18.0 is a drop-in replacement for 3.16.0.** No rule, -macro, provider or attribute changed; every commit between the two tags is one +**For a bzlmod consumer, 3.19.0 is a drop-in replacement for 3.16.0.** No rule, +macro, provider or attribute changed; every commit between the tags is one documented fix. The one removal is the WORKSPACE entry point, which no supported Bazel can load anyway. See [CHANGELOG.md](./CHANGELOG.md). ## Installation -> **Git pre-release only.** Pick one of the two override methods below. Once the -> Bazel Central Registry entry lands, delete the override and keep the -> `bazel_dep`; nothing else changes. - -### Pinned to a release tag - -```starlark -bazel_dep(name = "rules_erlang", version = "3.18.0") - -archive_override( - module_name = "rules_erlang", - urls = ["https://github.com/bazelverse/rules_erlang/archive/refs/tags/3.18.0.tar.gz"], - strip_prefix = "rules_erlang-3.18.0", - integrity = "sha256-...", -) -``` - -To get the `integrity` value, run the build once with the attribute omitted. -Bazel fetches the archive and prints a warning containing the hash it computed; -paste that in. A *wrong* value fails the build and names the expected one, so -either route gets you there. Do not leave it out permanently: without it the -archive is re-fetched unverified. - -### Pinned to a commit +> **Git pre-release only.** The Bazel Central Registry entry is pending an +> ownership transfer, so `bazel_dep` alone will not resolve and you need an +> override. Once the entry lands, delete the override and keep the `bazel_dep`; +> nothing else changes. ```starlark -bazel_dep(name = "rules_erlang", version = "3.18.0") +bazel_dep(name = "rules_erlang", version = "3.19.0") git_override( module_name = "rules_erlang", remote = "https://github.com/bazelverse/rules_erlang.git", - commit = "0000000000000000000000000000000000000000", + commit = "0000000000000000000000000000000000000000", # 3.19.0 ) ``` -`commit` takes a full SHA. Use this to track work that has no tag yet. Prefer -the tag form for anything you ship: it is a fixed archive plus a checksum, -rather than a repository that has to stay reachable and re-clone. +`commit` takes a full SHA, not a tag. Pin the commit a release tag points at +rather than the tag itself: a SHA cannot be moved, and it is the same thing the +registry will hand you later. `git rev-list -n1 3.19.0` prints it. -The `version` in `bazel_dep` is still required either way. It is what the module -reports to the rest of the graph; the override decides what is actually fetched. +The `version` in `bazel_dep` is still required. It is what the module reports to +the rest of the graph; the override decides what is actually fetched. **Overrides only take effect in the root module.** If you depend on `rules_erlang` indirectly, through From eaad5e1d6f3e4d7abc6ab6c1aa586883ea6c3eff Mon Sep 17 00:00:00 2001 From: Marvin Hansen Date: Fri, 14 Aug 2026 16:47:34 +0800 Subject: [PATCH 4/4] feat: supply Erlang/OTP from a prebuilt binary release Adds prebuilt_erlang_from_http_archive and prebuilt_erlang_from_hex_builds, a third way to provide OTP alongside the existing external and internal paths. Neither existing path covers a downloadable prebuilt. external_erlang_from_path takes an absolute erlang_home that must already exist on whatever machine runs the action -- a host dependency, and unsatisfiable on a remote executor without baking OTP into the executor image. internal_erlang_from_http_archive accepts any URL but always feeds it to erlang_build, which runs ./configure && make; pointing it at a binary release does not skip the build, it fails, because there is nothing to configure. So a consumer who wanted a prebuilt OTP had to compile one anyway. Measured on the same target and machine with both caches defeated via --action_env, OTP 28.1 on linux/amd64: the from-source path was still compiling after five minutes and was abandoned; the prebuilt path completed in ~12s wall, of which the staging action was 3-8s. The download is 71 MB for a 171 MB installed tree. On relocation. An OTP installation is not relocatable -- ROOTDIR is written into the generated start scripts at install time, which is why a binary release ships an Install script at all. `Install -cross ` treats the current directory as the staging location while writing TARGET into the scripts, and that is exactly the split this rule needs: staging happens in a sandbox, but the install path is fixed and absolute. The rule fails with an explicit message when the archive has no executable Install at its root, because the likely mistake is passing an otp_src tarball, and the resulting error otherwise surfaces deep inside the action as a missing ./configure. On the output contract. erlang_prebuilt returns the same release_dir_tar and version_file that erlang_build does, and validates the same way -- unpack where the installation will actually live, run erl, and compare what it reports against the declared version rather than trusting the attribute. The toolchain cannot tell the two rules apart. On toolchain registration. A prebuilt installation registers under the existing :erlang_internal constraint rather than a new one. It is hermetic in the same sense an internally built one is: fetched and staged by the build, not discovered on the host. Reusing the constraint keeps the generated target names identical, so moving a repository between the source and prebuilt paths is a MODULE.bazel edit and nothing else -- existing toolchain_resolution_overrides and //:toolchain_major references keep resolving. On multiple architectures. Both tag classes take exec_compatible_with, appended to the generated toolchain's own exec_compatible_with. A repository can register one prebuilt per execution architecture and let toolchain resolution choose between them, which is what makes a prebuilt OTP useful for something other than the machine that happens to be building. prebuilt_erlang_from_hex_builds is a thin convenience over builds.hex.pm, which publishes a release per (arch, os) and lists each archive's sha256 in its builds.txt. The checksum a consumer pins is therefore one upstream already publishes, rather than one produced by mirroring the archive somewhere to compute it. Note that hex.pm publishes arm64 only for ubuntu-22.04 and ubuntu-24.04; the other distributions are amd64-only, and the os value has to match the runtime's libc. Verified against a monorepo whose Elixir build was switched to the prebuilt path: the OTP stages and erl runs correctly from the relocated install reporting 28.1, and five Elixir test suites compile and pass through it. This repository's own three tests pass (forced, not cached). Not covered: there is no test here exercising erlang_prebuilt. The test workspace registers its OTP toolchains by name, so adding one means a second registered installation and the resolution ambiguity that comes with it. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: Marvin Hansen --- CHANGELOG.md | 78 +++++++++++++++ MODULE.bazel | 2 +- bzlmod/extensions.bzl | 111 +++++++++++++++++---- private/erlang_build.bzl | 165 +++++++++++++++++++++++++++++++- repositories/BUILD_prebuilt.tpl | 68 +++++++++++++ repositories/erlang_config.bzl | 31 ++++++ 6 files changed, 432 insertions(+), 23 deletions(-) create mode 100644 repositories/BUILD_prebuilt.tpl diff --git a/CHANGELOG.md b/CHANGELOG.md index 73557f3..943200d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,83 @@ # Changelog +## 3.20.0 + +Adds a third way to supply Erlang/OTP: an already-compiled binary release. + +### Added + +- **`prebuilt_erlang_from_http_archive` and `prebuilt_erlang_from_hex_builds`** + (`bzlmod/extensions.bzl`, `repositories/erlang_config.bzl`, + new `repositories/BUILD_prebuilt.tpl`, new `erlang_prebuilt` in + `private/erlang_build.bzl`). + + Until now there were two ways to provide OTP, and neither covers a downloadable + prebuilt: + + - `external_erlang_from_path` takes an absolute `erlang_home` that must already + exist on whatever machine runs the action. That is a host dependency, which is + exactly what a hermetic build is trying not to have, and it cannot be satisfied + on a remote executor without baking OTP into the executor image. + - `internal_erlang_from_http_archive` accepts any URL, but always feeds it to + `erlang_build`, which runs `./configure && make`. Pointing it at a binary + release does not skip the build; it fails, because there is nothing to + configure. + + So a consumer who wanted a prebuilt OTP had to compile one anyway. The new tag + classes fetch an OTP *binary release* and stage it, with no configure or make + step. + + Measured on the same target, same machine, same forced re-execution + (`--action_env=CACHEBUST=...` to defeat both caches), OTP 28.1 on linux/amd64: + + | | | + |---|---| + | `internal_erlang_from_github_release` | still compiling after 5 min, abandoned | + | `prebuilt_erlang_from_hex_builds` | ~12s wall, staging action ~3-8s | + + The download is 71 MB for a 171 MB installed tree. + + `prebuilt_erlang_from_hex_builds` is a thin convenience over `builds.hex.pm`, + which publishes a release per `(arch, os)` and lists each archive's sha256 in + its `builds.txt`. The checksum a consumer pins is therefore one upstream already + publishes, rather than one produced by mirroring the archive somewhere to compute + it. + + ```python + erlang_config_ext.prebuilt_erlang_from_hex_builds( + name = "otp_28_1", + version = "28.1", + arch = "amd64", # or "arm64" + os = "ubuntu-24.04", # must match the runtime's libc + sha256 = "6f7a95250a83f999909cf64dc375fd08809eb057f61b24eecdf2b3b44fe621ab", + ) + ``` + + **On relocation.** An OTP installation is not relocatable: `ROOTDIR` is written + into the generated start scripts at install time. A binary release ships an + `Install` script precisely because of this, and `Install -cross ` treats + the current directory as the staging location while writing `TARGET` into the + scripts. That is the split this rule needs, since staging happens in a sandbox + and the install path is fixed and absolute. The rule fails with an explicit + message if the archive has no executable `Install` at its root, because the + likely mistake is passing an `otp_src` tarball, and the error from doing that + otherwise surfaces deep inside the action. + + **On toolchain registration.** A prebuilt installation registers under the + existing `:erlang_internal` constraint rather than a new one. It is hermetic in + the same sense an internally built one is -- fetched and staged by the build, + not discovered on the host -- and reusing the constraint keeps the generated + target names identical. Moving a repository between the source and prebuilt + paths is therefore a `MODULE.bazel` edit and nothing else: existing + `toolchain_resolution_overrides` and `//:toolchain_major` references keep + resolving. + + **On multiple architectures.** Both tag classes take `exec_compatible_with`, + whose values are appended to the generated toolchain's `exec_compatible_with`. + A repository can register one prebuilt per execution architecture and let + toolchain resolution choose, which is what makes a prebuilt OTP usable for + something other than the machine that happens to be building. + ## 3.19.0 A single fix, for a bug 3.18.0 introduced. diff --git a/MODULE.bazel b/MODULE.bazel index 25157ec..201f839 100644 --- a/MODULE.bazel +++ b/MODULE.bazel @@ -1,6 +1,6 @@ module( name = "rules_erlang", - version = "3.19.0", + version = "3.20.0", bazel_compatibility = [">=8.0.0"], ) diff --git a/bzlmod/extensions.bzl b/bzlmod/extensions.bzl index d8962fb..dc6d58b 100644 --- a/bzlmod/extensions.bzl +++ b/bzlmod/extensions.bzl @@ -1,7 +1,18 @@ load( - ":hex_pm.bzl", - "hex_package_info", - "satisfies", + "//private:xref_runner_sources.bzl", + "xref_runner_sources", +) +load( + "//repositories:erlang_config.bzl", + "INSTALLATION_TYPE_EXTERNAL", + "INSTALLATION_TYPE_INTERNAL", + "INSTALLATION_TYPE_PREBUILT", + _erlang_config_rule = "erlang_config", +) +load( + "//tools:erlang.bzl", + "DEFAULT_ERLANG_SHA256", + "DEFAULT_ERLANG_VERSION", ) load( ":erlang_package.bzl", @@ -11,27 +22,17 @@ load( "log", "without_requirement", ) +load( + ":hex_pm.bzl", + "hex_package_info", + "satisfies", +) load( ":semver.bzl", "compatible", "lt", "version_from_string", ) -load( - "//private:xref_runner_sources.bzl", - "xref_runner_sources", -) -load( - "//repositories:erlang_config.bzl", - "INSTALLATION_TYPE_EXTERNAL", - "INSTALLATION_TYPE_INTERNAL", - _erlang_config_rule = "erlang_config", -) -load( - "//tools:erlang.bzl", - "DEFAULT_ERLANG_SHA256", - "DEFAULT_ERLANG_VERSION", -) def _erlang_config(ctx): types = {} @@ -44,6 +45,7 @@ def _erlang_config(ctx): extra_configure_optss = {} post_configure_cmdss = {} extra_make_optss = {} + extra_exec_constraintss = {} owners_by_name = {} for mod in ctx.modules: @@ -105,6 +107,46 @@ def _erlang_config(ctx): extra_make_optss[erlang.name] = erlang.extra_make_opts owners_by_name[erlang.name] = mod + for erlang in mod.tags.prebuilt_erlang_from_http_archive: + if erlang.name in types: + fail("{} declares an erlang installation named {}, but the name is already used by {}".format( + mod.name, + erlang.name, + owners_by_name[erlang.name].name, + )) + types[erlang.name] = INSTALLATION_TYPE_PREBUILT + versions[erlang.name] = erlang.version + urls[erlang.name] = erlang.url + strip_prefixs[erlang.name] = erlang.strip_prefix + sha256s[erlang.name] = erlang.sha256 + extra_exec_constraintss[erlang.name] = erlang.exec_compatible_with + owners_by_name[erlang.name] = mod + + for erlang in mod.tags.prebuilt_erlang_from_hex_builds: + if erlang.name in types: + fail("{} declares an erlang installation named {}, but the name is already used by {}".format( + mod.name, + erlang.name, + owners_by_name[erlang.name].name, + )) + + # builds.hex.pm publishes an OTP binary release per (arch, os) with the sha256 of + # each archive listed alongside it in builds.txt, so a consumer pins the checksum + # upstream already publishes rather than mirroring the archive to compute one. + url = "https://builds.hex.pm/builds/otp/{arch}/{os}/OTP-{v}.tar.gz".format( + arch = erlang.arch, + os = erlang.os, + v = erlang.version, + ) + + types[erlang.name] = INSTALLATION_TYPE_PREBUILT + versions[erlang.name] = erlang.version + urls[erlang.name] = url + strip_prefixs[erlang.name] = "OTP-{}".format(erlang.version) + sha256s[erlang.name] = erlang.sha256 + extra_exec_constraintss[erlang.name] = erlang.exec_compatible_with + owners_by_name[erlang.name] = mod + _erlang_config_rule( name = "erlang_config", rules_erlang_workspace = "@rules_erlang", @@ -118,6 +160,7 @@ def _erlang_config(ctx): extra_configure_optss = extra_configure_optss, post_configure_cmdss = post_configure_cmdss, extra_make_optss = extra_make_optss, + extra_exec_constraintss = extra_exec_constraintss, ) external_erlang_from_path = tag_class(attrs = { @@ -152,12 +195,44 @@ internal_erlang_from_github_release = tag_class(attrs = { "extra_make_opts": attr.string_list(), }) +# Any OTP *binary release* -- an archive with an `Install` script at its root, as opposed to +# an otp_src source tarball. `exec_compatible_with` lets a repository register one prebuilt per +# execution architecture and let toolchain resolution choose between them. +prebuilt_erlang_from_http_archive = tag_class(attrs = { + "name": attr.string(), + "version": attr.string(), + "url": attr.string(), + "strip_prefix": attr.string(), + "sha256": attr.string(), + "exec_compatible_with": attr.string_list(), +}) + +# Convenience wrapper over builds.hex.pm, which publishes an OTP binary release per +# (arch, os) together with its sha256 in builds.txt -- so `sha256` here is a value copied +# from upstream rather than one computed by mirroring the archive. +prebuilt_erlang_from_hex_builds = tag_class(attrs = { + "name": attr.string(), + "version": attr.string(), + "arch": attr.string( + default = "amd64", + doc = "Architecture segment as published by builds.hex.pm, e.g. amd64 or arm64.", + ), + "os": attr.string( + default = "ubuntu-24.04", + doc = "OS segment as published by builds.hex.pm. Must match the runtime's libc.", + ), + "sha256": attr.string(), + "exec_compatible_with": attr.string_list(), +}) + erlang_config = module_extension( implementation = _erlang_config, tag_classes = { "external_erlang_from_path": external_erlang_from_path, "internal_erlang_from_http_archive": internal_erlang_from_http_archive, "internal_erlang_from_github_release": internal_erlang_from_github_release, + "prebuilt_erlang_from_http_archive": prebuilt_erlang_from_http_archive, + "prebuilt_erlang_from_hex_builds": prebuilt_erlang_from_hex_builds, }, ) diff --git a/private/erlang_build.bzl b/private/erlang_build.bzl index 86a8d34..0448323 100644 --- a/private/erlang_build.bzl +++ b/private/erlang_build.bzl @@ -1,12 +1,12 @@ +load( + "@bazel_skylib//rules:common_settings.bzl", + "BuildSettingInfo", +) load( "@bazel_tools//tools/build_defs/hash:hash.bzl", "sha256", "tools", ) -load( - "@bazel_skylib//rules:common_settings.bzl", - "BuildSettingInfo", -) load( "//:util.bzl", "BEGINS_WITH_FUN", @@ -231,6 +231,163 @@ erlang_build = rule( }, ) +def _erlang_prebuilt_impl(ctx): + """Stage an already-compiled OTP release instead of building one. + + Same output contract as erlang_build -- a release_dir_tar that unpacks into install_path, + and a version_file -- so the toolchain cannot tell the two apart. What differs is that no + configure/make runs: the archive is an OTP *binary release*, which ships an `Install` + script exactly because such a release has to be re-pointed at wherever it ends up. + + OTP installations are not relocatable; ROOTDIR is baked into the generated start scripts. + `Install -cross ` is the supported way out: it treats the current directory as the + staging location and writes TARGET into the scripts, which is precisely the split this rule + needs, since staging happens in a sandbox and the install path is fixed and absolute. + """ + (_, _, filename) = ctx.attr.url.rpartition("/") + downloaded_archive = ctx.actions.declare_file(filename) + release_dir_tar = ctx.actions.declare_file(ctx.label.name + "_release.tar") + version_file = ctx.actions.declare_file(ctx.label.name + "_version") + + if not ctx.attr.install_prefix.startswith("/"): + # As with erlang_build: not relocatable, so this has to be absolute. + fail("install_prefix must be absolute") + install_path = path_join(ctx.attr.install_prefix, ctx.label.name) + erlang_home = path_join(install_path, "lib", "erlang") + + ctx.actions.run_shell( + inputs = [], + outputs = [downloaded_archive], + command = """set -euo pipefail + +curl -L "{archive_url}" -o {archive_path} +""".format( + archive_url = ctx.attr.url, + archive_path = downloaded_archive.path, + ), + mnemonic = "OTP", + progress_message = "Downloading {}".format(ctx.attr.url), + ) + + sha256file = sha256(ctx, downloaded_archive) + + # Same reasoning as erlang_build: --strip-components rather than --transform, because + # macOS ships bsdtar. A binary release wraps everything in one top-level directory. + strip_components = "--strip-components=1" if ctx.attr.strip_prefix != "" else "" + + ctx.actions.run_shell( + inputs = [downloaded_archive, sha256file], + outputs = [release_dir_tar], + command = """set -euo pipefail + +if [ -n "{sha256}" ]; then + if [ "{sha256}" != "$(cat "{sha256file}")" ]; then + echo "ERROR: Checksum mismatch. $(basename "{archive_path}") $(cat "{sha256file}") != {sha256}" + exit 1 + fi +fi + +ABS_RELEASE_DIR_TAR=$PWD/{release_path} +ABS_STAGE_DIR="$(mktemp -d)" + +# The tar is laid out so that unpacking it into install_path yields install_path/lib/erlang, +# which is what erlang_home points at -- matching what erlang_build produces via `make install`. +mkdir -p "$ABS_STAGE_DIR/lib/erlang" +tar --extract \\ + {strip_components} \\ + --file "{archive_path}" \\ + --directory "$ABS_STAGE_DIR/lib/erlang" + +cd "$ABS_STAGE_DIR/lib/erlang" + +if [ ! -x ./Install ]; then + echo "ERROR: $(pwd) has no executable Install script." + echo " erlang_prebuilt expects an OTP binary release (the layout published at" + echo " builds.hex.pm), not an otp_src source tarball -- use erlang_build for source." + exit 1 +fi + +# -cross: bake {erlang_home} into the start scripts while operating on this staged copy. +./Install -cross -minimal "{erlang_home}" + +cd "$ABS_STAGE_DIR" +tar --create \\ + --file "$ABS_RELEASE_DIR_TAR" \\ + * +""".format( + sha256 = ctx.attr.sha256v, + sha256file = sha256file.path, + archive_path = downloaded_archive.path, + strip_components = strip_components, + release_path = release_dir_tar.path, + erlang_home = erlang_home, + ), + use_default_shell_env = True, + mnemonic = "OTP", + progress_message = "Staging prebuilt otp {}".format(ctx.attr.version), + ) + + # Identical validation to erlang_build: unpack where it will actually live and ask the + # runtime its own version, rather than trusting the version attribute. + ctx.actions.run_shell( + inputs = [release_dir_tar], + outputs = [version_file], + command = """set -euo pipefail + +mkdir -p "{install_path}" +tar --extract \\ + --directory "{install_path}" \\ + --file {erlang_release_tar} + +{begins_with_fun} +V=$("{erlang_home}"/bin/{query_erlang_version}) +if ! beginswith "{erlang_version}" "$V"; then +echo "Erlang version mismatch (Expected {erlang_version}, found $V)" +exit 1 +fi + +echo "$V" >> {version_file} +""".format( + install_path = install_path, + begins_with_fun = BEGINS_WITH_FUN, + query_erlang_version = QUERY_ERL_VERSION, + erlang_version = ctx.attr.version, + erlang_home = erlang_home, + erlang_release_tar = release_dir_tar.path, + version_file = version_file.path, + ), + mnemonic = "OTP", + progress_message = "Validating prebuilt otp at {}".format(erlang_home), + ) + + return [ + DefaultInfo( + files = depset([ + release_dir_tar, + version_file, + ]), + ), + OtpInfo( + version = ctx.attr.version, + release_dir_tar = release_dir_tar, + install_path = install_path, + erlang_home = erlang_home, + version_file = version_file, + ), + ] + +erlang_prebuilt = rule( + implementation = _erlang_prebuilt_impl, + attrs = { + "version": attr.string(mandatory = True), + "url": attr.string(mandatory = True), + "strip_prefix": attr.string(), + "sha256v": attr.string(), + "install_prefix": attr.string(default = DEFAULT_INSTALL_PREFIX), + "sha256": tools["sha256"], + }, +) + def _erlang_external_impl(ctx): erlang_home = ctx.attr.erlang_home if erlang_home == "": diff --git a/repositories/BUILD_prebuilt.tpl b/repositories/BUILD_prebuilt.tpl new file mode 100644 index 0000000..310a537 --- /dev/null +++ b/repositories/BUILD_prebuilt.tpl @@ -0,0 +1,68 @@ +# This file is generated by rules_erlang via the erlang_config macro + +load( + "%{RULES_ERLANG_WORKSPACE}//private:erlang_build.bzl", + "erlang_prebuilt", +) +load( + "%{RULES_ERLANG_WORKSPACE}//tools:erlang_toolchain.bzl", + "erlang_toolchain", +) + +erlang_prebuilt( + name = "otp-%{ERLANG_NAME}", + version = "%{ERLANG_VERSION}", + url = "%{URL}", + strip_prefix = "%{STRIP_PREFIX}", + sha256v = "%{SHA_256}", +) + +erlang_toolchain( + name = "erlang_%{ERLANG_MAJOR}_%{ERLANG_MINOR}_toolchain", + otp = ":otp-%{ERLANG_NAME}", + visibility = ["//visibility:public"], +) + +# A prebuilt OTP is hermetic in the same sense an internally built one is -- it is fetched and +# staged by the build rather than discovered on the host -- so it registers under the same +# :erlang_internal constraint. That keeps the toolchain target names identical to the +# from-source path, so switching a repository between them is a MODULE.bazel edit and nothing +# else: existing toolchain_resolution_overrides and //...:toolchain_major references still +# resolve. +toolchain( + name = "toolchain_major", + exec_compatible_with = [ + "//:erlang_internal", +%{EXTRA_EXEC_CONSTRAINTS} ], + target_compatible_with = [ + "//:erlang_%{ERLANG_MAJOR}", + ], + toolchain = ":erlang_%{ERLANG_MAJOR}_%{ERLANG_MINOR}_toolchain", + toolchain_type = "%{RULES_ERLANG_WORKSPACE}//tools:toolchain_type", + visibility = ["//visibility:public"], +) + +alias( + name = "toolchain", + actual = "toolchain_major", + visibility = ["//visibility:public"], +) + +toolchain( + name = "toolchain_major_minor", + exec_compatible_with = [ + "//:erlang_internal", +%{EXTRA_EXEC_CONSTRAINTS} ], + target_compatible_with = [ + "//:erlang_%{ERLANG_MAJOR}_%{ERLANG_MINOR}", + ], + toolchain = ":erlang_%{ERLANG_MAJOR}_%{ERLANG_MINOR}_toolchain", + toolchain_type = "%{RULES_ERLANG_WORKSPACE}//tools:toolchain_type", + visibility = ["//visibility:public"], +) + +alias( + name = "toolchain2", + actual = "toolchain_major_minor", + visibility = ["//visibility:public"], +) diff --git a/repositories/erlang_config.bzl b/repositories/erlang_config.bzl index a2808ad..43038e6 100644 --- a/repositories/erlang_config.bzl +++ b/repositories/erlang_config.bzl @@ -12,6 +12,12 @@ _ERLANG_VERSION_UNKNOWN = "UNKNOWN" INSTALLATION_TYPE_EXTERNAL = "external" INSTALLATION_TYPE_INTERNAL = "internal" +# An OTP binary release fetched and staged by the build rather than compiled from source. +# Hermetic in the same sense as "internal" -- not discovered on the host -- but with no +# configure/make step, which is the entire point: it removes a multi-minute action from +# every cold build. +INSTALLATION_TYPE_PREBUILT = "prebuilt" + def _parse_maybe_semver(version_string): parts = version_string.split(".", 2) if len(parts) > 1: @@ -46,6 +52,7 @@ def _impl(repository_ctx): extra_configure_opts = repository_ctx.attr.extra_configure_optss.get(name, []), post_configure_cmds = repository_ctx.attr.post_configure_cmdss.get(name, []), extra_make_opts = repository_ctx.attr.extra_make_optss.get(name, []), + extra_exec_constraints = repository_ctx.attr.extra_exec_constraintss.get(name, []), ) # If we intentionally skipped probing a host-installed OTP, still provide @@ -68,6 +75,29 @@ def _impl(repository_ctx): }, False, ) + elif props.type == INSTALLATION_TYPE_PREBUILT: + extra = getattr(props, "extra_exec_constraints", []) + repository_ctx.template( + "{}/BUILD.bazel".format(name), + Label("//repositories:BUILD_prebuilt.tpl"), + { + "%{ERLANG_NAME}": name, + "%{ERLANG_VERSION}": props.version, + "%{URL}": props.url, + "%{STRIP_PREFIX}": props.strip_prefix or "", + "%{SHA_256}": props.sha256 or "", + "%{ERLANG_MAJOR}": props.major, + "%{ERLANG_MINOR}": props.minor, + "%{RULES_ERLANG_WORKSPACE}": rules_erlang_workspace, + # Added to exec_compatible_with so a repository can register one prebuilt + # per execution architecture and let toolchain resolution pick between them. + "%{EXTRA_EXEC_CONSTRAINTS}": "".join([ + ' "%s",\n' % c + for c in extra + ]), + }, + False, + ) else: repository_ctx.template( "{}/BUILD.bazel".format(name), @@ -131,6 +161,7 @@ erlang_config = repository_rule( "extra_configure_optss": attr.string_list_dict(), "post_configure_cmdss": attr.string_list_dict(), "extra_make_optss": attr.string_list_dict(), + "extra_exec_constraintss": attr.string_list_dict(), }, environ = [ ERLANG_HOME_ENV_VAR,