Repository navigation
chore: speedup semver check using shared target directory - #26598
Merged
neilconway merged 1 commit intoOct 23, 2026
Merged
Conversation
cargo-semver-checks builds the rustdoc for each changed crate twice, once for the PR and once for the baseline, and does each build in its own placeholder project. Unless CARGO_TARGET_DIR is set, each of those projects gets its own target directory, so every build compiled all of the crate's dependencies, such as arrow, parquet, and sqlparser, from scratch. In CI logs, each crate's baseline build takes as long as its current build. changed_crates.sh now sets CARGO_TARGET_DIR to semver-checks-shared inside the workspace's target directory, so these builds compile each dependency once. DataFusion's own crates are still built separately for the PR and the baseline, since they come from different source trees. The cargo-semver-checks maintainer suggests this setting for workspaces that check many crates (cargo-semver-checks#401). Measured locally against origin/main with cargo-semver-checks 0.51.0, the version CI installs, and one build job at a time: - datafusion-optimizer: 67s before, 39s after - datafusion-physical-plan, datafusion, and datafusion-physical-optimizer: 327s before, 131s after (CPU time 340s before, 129s after) Both versions report the same failure for a deliberate breaking change (making a pub fn in datafusion-physical-plan pub(crate)), and the shared directory uses less disk: 2.0 GB, against 3.4 GB before, for the three-crate run.
github-actions
Bot
added
the
development-process
kumarUjjawal
approved these changes
Oct 23, 2026
kumarUjjawal
left a comment
kumarUjjawal
left a comment
Contributor
There was a problem hiding this comment.
Thanks @neilconway
LGTM 🚀
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
For every changed crate,
cargo-semver-checkscompiles two versions of it (baseline version vs. PR version). By default, each build uses its own target directory. For example, a PR touching three crates will result in six from-scratch compilations.We can improve this by setting
CARGO_TARGET_DIR, so that all of these builds use a shared target directory. This means that unmodified dependencies need only be built once.This improves semver check runtime by ~1.8-3x, depending on the nature of the PR, and also reduces disk space consumption.
What changes are included in this PR?
See above.
What is the testing strategy for this PR?
Tested locally, dev tooling change only.
Are there any user-facing changes?
No.