chore(ci): shard and only run perf benchmarks on impacted crates in PRs#2191
Draft
ekump wants to merge 4 commits into
Draft
chore(ci): shard and only run perf benchmarks on impacted crates in PRs#2191ekump wants to merge 4 commits into
ekump wants to merge 4 commits into
Conversation
Contributor
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 1a3369d | Docs | Datadog PR Page | Give us feedback! |
Contributor
📚 Documentation Check Results📦
|
Contributor
Clippy Allow Annotation ReportComparing clippy allow annotations between branches:
Summary by Rule
Annotation Counts by File
Annotation Stats by Crate
About This ReportThis report tracks Clippy allow annotations for specific rules, showing how they've changed in this PR. Decreasing the number of these annotations generally improves code quality. |
Contributor
🔒 Cargo Deny Results📦
|
Contributor
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
ekump
force-pushed
the
ekump/APMSP-3628-only-run-perf-bench-for-impacted-crates
branch
from
239b7c7 to
db0efdc
Compare
ekump
changed the title
ekump
force-pushed
the
ekump/APMSP-3628-only-run-perf-bench-for-impacted-crates
branch
from
790a5f5 to
1a3369d
Compare
yannham
approved these changes
Jul 3, 2026
yannham
left a comment
yannham
left a comment
Contributor
There was a problem hiding this comment.
So we get parallel and "differential" benches, nice 👍
| allow_failure: true | ||
| script: | ||
| - git fetch --no-tags origin main | ||
| - (cd .github/actions && cargo build --release -p crates-reporter) |
Contributor
There was a problem hiding this comment.
It's more of a nitpick / future work, but would there be a way to distribute a binary of crates-reporter instead? I suppose it doesn't change often. But sometimes getting a binary in a job is just so annoying that maybe building from source is simpler 🤷
| message "Benchmarking selected crates: ${BENCH_PACKAGES}" | ||
| cargo bench "${package_args[@]}" "${feature_args[@]}" -- --warm-up-time 1 --measurement-time 5 --sample-size=200 | ||
| else | ||
| cargo bench --workspace --features libdd-crashtracker/benchmarking,libdd-sampling/v04_span,libdd-sampling/bench-internals,libdd-trace-utils/bench-internals -- --warm-up-time 1 --measurement-time 5 --sample-size=200 |
Contributor
There was a problem hiding this comment.
Nit: this is probably simpler with this one liner, but I wonder if we shouldn't generate a BENCH_PACKAGES unconditionally (just putting all the crates that are known to bench when it's empty) and then use a single code path for the cargo bench command and package features. Otherwise there are two places where we define which features a specific crate needs for benchmarking (this line and in bench_features_for_crate), and they could disagree/drift.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Speeds up the performance benchmark job that runs in Gitlab by doing 2 things:
Motivation
What inspired you to submit this pull request?
Additional Notes
The actual performance improvements should be much bigger once this is merged. The candidate benchmarks are filtering to only impacted crates, but the baseline references main and is still running the full suite.
How to test the change?
Temporarily modified the branch to modify
libdd-trace-utilsand saw that it ran for only impacted crates on the candidate benchmarks. https://gitlab.ddbuild.io/DataDog/apm-reliability/libdatadog/-/pipelines/122493383The script that posts the comment to PRs of benchmarks results has been broken prior to this PR. That will need to be fixed separately before we can fully evaluate this PR works as expected.