You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Compare pull requests against a baseline measured on the same machine - #1894
A pull request is measured against the daily benchmark run at the commit it
branched from. That run came from another machine. The runner pools hold more
than one processor model per vendor and are shared with whatever else is
scheduled there, so the difference between two machines reads as a difference
between two commits.
PR #1893 touches one file of CI tooling and cannot alter a benchmark. It was
reported as a 1.58x regression on hyperv-ws2025 / intel, because its baseline
landed on a processor the rest of the week did not:
run
virtq_readwrite/slot_pool_segmented/262144
processor
2 Oct
43.2k
Model 106
3 Oct
43.0k
Model 106
4 Oct
45.2k
Model 106
5 Oct
43.1k
Model 106
6 Oct, the baseline
37.6k
Model 143
the pull request
59.5k
Model 106
Cause
Two machines, not two commits. Of the six configurations, two draw from pools
holding more than one processor model, where the models differ by 10% and 11.9%
while runs on the same model differ by 2% and 4.4%.
Shared tenancy moves the rest. payload_allocation/slot_pool_segmented/262144
sat at 524, 526, 526, 533 and 593 ns across the week on one processor model,
and the same commit measured 835 ns on the pull request's runner.
Criterion's confidence interval does not see either. It measures the samples
within one run, which vary by 0.90%, while a run measured again the next day
moves 7.65%, more than the interval allows in 111 of 114 benchmarks.
What this does
cargo ci bench --baseline-ref <COMMIT> measures that commit before the one
checked out, saving its results as the baseline criterion then compares
against. The two sit in one criterion directory, so a report reads the
comparison the run carries and names no baseline of its own.
The baseline is built in a git worktree, so the checkout in hand is never moved
and a run that fails leaves nothing to restore. Each benchmark binary resolves
its guests relative to its own source tree, so the baseline reads the worktree
while the current tree keeps its own, with no swapping between passes.
The guests the baseline needs are built by the job that already holds the
toolchain for them. HEAD^1 names the branch point because the checkout is the
merge its validation builds, and both jobs assert that before relying on it,
since on anything else HEAD^1 is the previous commit and would measure
against it without saying so.
Also here
criterion-markdown 0.2.2. A report shows criterion's typical estimate, the
regression slope wherever linear sampling fits one, while the change beside it
read the mean. The mean weights every sample alike and so carries the warm-up
the slope leaves behind: slot_pool/alloc_dealloc_1500 reported 1.56x slower
while its slope held 7% faster, from 32 samples that ran cold before the rest
settled.
Verified
The pass and its comparison run end to end, leaving base, new and change
in one directory for the report to read, and the worktree removed after it. cargo ci bench without --baseline-ref is unchanged, which is what the daily
runs use.
The benchmark report on this pull request is the first one measured this way.
A report shows criterion's typical estimate, the regression slope wherever
linear sampling fits one, while the change beside it read the mean. The mean
weights every sample alike and so carries the warm-up the slope leaves behind,
which read as a regression on benchmarks whose first samples ran cold.
0.2.2 compares the estimate it displays, and holds a verdict back until the
whole confidence interval clears its threshold.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
A pull request was measured against the daily run its branch point carried.
That run came from another machine, and the pools hold more than one processor
model per vendor and are shared with whatever else is scheduled there. The
difference between two runners read as a difference between two commits:
one report announced a 1.58x regression for a change that touches only CI
tooling, from a baseline that happened to land on faster silicon.
`cargo ci bench --baseline-ref` measures a second commit before the one checked
out, in a git worktree so the checkout in hand is never moved and a run that
fails leaves nothing to restore. Each benchmark binary resolves its guests
relative to its own source tree, so the baseline reads the worktree and the
current tree keeps its own, and criterion compares the two from one directory.
The guests the baseline needs are built by the job that already has the
toolchain for them. `HEAD^1` names the branch point because the checkout is
the merge its validation builds, which both jobs assert before relying on.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
A commit may add a benchmark the one it is measured against never had, which
criterion's strict `--baseline` treats as a failure, so every pull request that
extends the suite would fail its benchmark job. The comparison reads leniently
now, and the baseline it is about to write is cleared first, so a copy left by
an earlier run cannot stand in for a benchmark this one could not measure.
A configuration named by a relative path was read after entering the worktree,
from whatever the commit being measured carried at the same path. It is
resolved before either pass runs.
A pre-built `--binary` holds one commit and cannot represent two, so it is
refused alongside a baseline rather than measuring the same executable twice.
Worktrees are pruned before one is added, since `cargo clean` takes the
directory while leaving the registration behind.
`HEAD^1` is the commit a pull request merges into rather than where it
branched, which is what the comparison wants and what the comments now say.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
`HEAD^1` is the commit a pull request merges into, which carries whatever
landed on the branch it targets since the pull request left it. Those commits
are not the pull request's to answer for.
`HEAD^1...HEAD^2` names where the two parents of the validation merge branched
apart, which is where the pull request did. `git checkout` and `git diff`
already read a revision that way. `git worktree add` takes a reference rather
than a revision, so the naming is done before it is handed a commit.
The form also says when it does not hold: without a second parent there is
nothing to branch apart from, where `HEAD^1` would quietly name the commit
before.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
A report named its baseline from the second set of results it read. A run that
measures its own baseline reads no second set, so the report said only which
commit it measured and left the reader to guess the other.
The run records the commit it measured against in its manifest, beside what it
measured and the machine it ran on, and the report names it from there.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
The reason will be displayed to describe this comment to others. Learn more.
Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.
Note
This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.
Quote guest paths containing spaces in baseline copying
Justfile:467
A guest directory containing spaces is split by the recipe shell, so --baseline-guests receives a truncated path and guest copying fails. The workflow's quotes protect the argument passed to just, but not this interpolation. Quote the path here too.
Use the selected baseline name when rendering benchmark reports
src/hyperlight_ci/src/bench.rs:106
A non-default --baseline-name saves the paired baseline somewhere bench-report does not read. Its renderer defaults to base/estimates.json, so the report either omits comparisons or uses an unrelated existing base, even though the manifest names the paired commit. Persist the selected dataset name and pass it to the renderer, or keep paired runs fixed to base.
Allow missing benchmarks during lenient baseline selection
src/hyperlight_ci/src/bench.rs:155
If --config-file allowlists a benchmark introduced by the current commit, the baseline pass fails in BenchConfig::select() before measuring the current tree. Selection rejects unmatched patterns and empty filtered suites, so Criterion's --baseline-lenient cannot help. Allow absent benchmarks and an empty selection in the baseline pass while retaining strict validation for the current suite.
A benchmark may be grouped under the name a baseline is saved under, and
clearing the baseline took the group with it: its benchmarks, their last run
and every other baseline saved beside them.
Measurements tell the two apart. A saved baseline holds them, a group holds the
benchmarks below it, so the group is walked into rather than removed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Jorge Prendes <jorge.prendes@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A-testingRelated to tests or testing infrastructureC-bugFor PRs that fix bugs
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A pull request is measured against the daily benchmark run at the commit it
branched from. That run came from another machine. The runner pools hold more
than one processor model per vendor and are shared with whatever else is
scheduled there, so the difference between two machines reads as a difference
between two commits.
PR #1893 touches one file of CI tooling and cannot alter a benchmark. It was
reported as a 1.58x regression on
hyperv-ws2025 / intel, because its baselinelanded on a processor the rest of the week did not:
virtq_readwrite/slot_pool_segmented/262144Cause
Two machines, not two commits. Of the six configurations, two draw from pools
holding more than one processor model, where the models differ by 10% and 11.9%
while runs on the same model differ by 2% and 4.4%.
Shared tenancy moves the rest.
payload_allocation/slot_pool_segmented/262144sat at 524, 526, 526, 533 and 593 ns across the week on one processor model,
and the same commit measured 835 ns on the pull request's runner.
Criterion's confidence interval does not see either. It measures the samples
within one run, which vary by 0.90%, while a run measured again the next day
moves 7.65%, more than the interval allows in 111 of 114 benchmarks.
What this does
cargo ci bench --baseline-ref <COMMIT>measures that commit before the onechecked out, saving its results as the baseline criterion then compares
against. The two sit in one criterion directory, so a report reads the
comparison the run carries and names no baseline of its own.
The baseline is built in a git worktree, so the checkout in hand is never moved
and a run that fails leaves nothing to restore. Each benchmark binary resolves
its guests relative to its own source tree, so the baseline reads the worktree
while the current tree keeps its own, with no swapping between passes.
The guests the baseline needs are built by the job that already holds the
toolchain for them.
HEAD^1names the branch point because the checkout is themerge its validation builds, and both jobs assert that before relying on it,
since on anything else
HEAD^1is the previous commit and would measureagainst it without saying so.
Also here
criterion-markdown 0.2.2. A report shows criterion's typical estimate, the
regression slope wherever linear sampling fits one, while the change beside it
read the mean. The mean weights every sample alike and so carries the warm-up
the slope leaves behind:
slot_pool/alloc_dealloc_1500reported 1.56x slowerwhile its slope held 7% faster, from 32 samples that ran cold before the rest
settled.
Verified
The pass and its comparison run end to end, leaving
base,newandchangein one directory for the report to read, and the worktree removed after it.
cargo ci benchwithout--baseline-refis unchanged, which is what the dailyruns use.
The benchmark report on this pull request is the first one measured this way.