Skip to content

ci: Measure only SDK calls in CodSpeed using instrument-hooks - #119

Merged
khvn26 merged 2 commits into
feat/codspeedfrom
feat/codspeed-instrument-hooks
Oct 30, 2026
Merged

khvn26 merged 2 commits into
feat/codspeedfrom
feat/codspeed-instrument-hooks

Conversation

@khvn26

@khvn26 khvn26 commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

In this PR, we make sure that interpreter startup does not skew benchmark results, and affect the benchmark runtimes.

To achieve this, we basically whip up a quick pytest-codspeed port for Ruby, wrapping CodSpeed's instrument-hooks.

Run all benchmarks in one process, measuring each SDK call through a
native extension built on CodSpeed's instrument-hooks. Ruby start-up,
loading the SDK and building the client no longer count towards the
results.
@coderabbitai

coderabbitai Bot commented Oct 30, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

GCC 15, which ships with Ubuntu 26.04 runners, defaults to C23, where
zig.h's noreturn expands to an attribute that the generated core.c
places after the storage class, failing the build. Pin C11 like
pytest-codspeed does.
@codspeed

codspeed Bot commented Oct 30, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

🆕 15 new benchmarks
⏩ 15 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
🆕 getEnvironmentFlags[large] N/A 71.1 ms N/A
🆕 getEnvironmentFlags[medium] N/A 11.6 ms N/A
🆕 getEnvironmentFlags[small] N/A 1.3 ms N/A
🆕 getIdentityFlagsNoTraits[large] N/A 106.9 ms N/A
🆕 getIdentityFlagsNoTraits[medium] N/A 15.9 ms N/A
🆕 getIdentityFlagsNoTraits[small] N/A 2.4 ms N/A
🆕 getIdentityFlagsWithIdentityOverrides[large] N/A 108.4 ms N/A
🆕 getIdentityFlagsWithIdentityOverrides[medium] N/A 15.9 ms N/A
🆕 getIdentityFlagsWithIdentityOverrides[small] N/A 2.3 ms N/A
🆕 getIdentityFlagsWithTraits[large] N/A 114.7 ms N/A
🆕 getIdentityFlagsWithTraits[medium] N/A 16.7 ms N/A
🆕 getIdentityFlagsWithTraits[small] N/A 3 ms N/A
🆕 getIdentitySegmentsWithTraits[large] N/A 108.9 ms N/A
🆕 getIdentitySegmentsWithTraits[medium] N/A 14.2 ms N/A
🆕 getIdentitySegmentsWithTraits[small] N/A 2.9 ms N/A

Comparing feat/codspeed-instrument-hooks (d0b658f) with feat/codspeed (65967ee)

Open in CodSpeed

Footnotes

  1. 15 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@khvn26
khvn26 marked this pull request as ready for review October 30, 2026 17:21
@khvn26
khvn26 requested a review from a team as a code owner October 30, 2026 17:22
@khvn26
khvn26 requested review from emyller and removed request for a team October 30, 2026 17:22
@khvn26
khvn26 merged commit 867c9bb into feat/codspeed Oct 30, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants