From c5d57d9fe6fcc6a903cc2dd2418704cf4439f188 Mon Sep 17 00:00:00 2001 From: Neil Conway Date: Thu, 22 Oct 2026 13:51:38 -0400 Subject: [PATCH] ci: share one target directory across semver check builds cargo-semver-checks builds the rustdoc for each changed crate twice, once for the PR and once for the baseline, and does each build in its own placeholder project. Unless CARGO_TARGET_DIR is set, each of those projects gets its own target directory, so every build compiled all of the crate's dependencies, such as arrow, parquet, and sqlparser, from scratch. In CI logs, each crate's baseline build takes as long as its current build. changed_crates.sh now sets CARGO_TARGET_DIR to semver-checks-shared inside the workspace's target directory, so these builds compile each dependency once. DataFusion's own crates are still built separately for the PR and the baseline, since they come from different source trees. The cargo-semver-checks maintainer suggests this setting for workspaces that check many crates (cargo-semver-checks#401). Measured locally against origin/main with cargo-semver-checks 0.51.0, the version CI installs, and one build job at a time: - datafusion-optimizer: 67s before, 39s after - datafusion-physical-plan, datafusion, and datafusion-physical-optimizer: 327s before, 131s after (CPU time 340s before, 129s after) Both versions report the same failure for a deliberate breaking change (making a pub fn in datafusion-physical-plan pub(crate)), and the shared directory uses less disk: 2.0 GB, against 3.4 GB before, for the three-crate run. --- ci/scripts/changed_crates.sh | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/ci/scripts/changed_crates.sh b/ci/scripts/changed_crates.sh index be8c48b26ea11..96ca51b50832c 100755 --- a/ci/scripts/changed_crates.sh +++ b/ci/scripts/changed_crates.sh @@ -74,7 +74,15 @@ cmd_semver_check() { args+=(--package "$pkg") done - cargo semver-checks --baseline-rev "$base_ref" "${args[@]}" + # cargo-semver-checks builds the rustdoc for each crate, current and + # baseline, in a separate placeholder project. Each project has its own + # target directory unless CARGO_TARGET_DIR is set, so a shared directory + # lets those builds reuse the compiled dependencies. + local target_dir + target_dir=$(cargo metadata --no-deps --format-version 1 | jq -r .target_directory) + + CARGO_TARGET_DIR="${target_dir}/semver-checks-shared" \ + cargo semver-checks --baseline-rev "$base_ref" "${args[@]}" } # ── main ────────────────────────────────────────────────────────────