Repository navigation
Reduce CI startup overhead and stabilize test helpers - #299
Conversation
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: fdf392e | Docs | View more details | Give us feedback! |
|
/datadog autotest review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Codex Review: Didn't find any major issues. Breezy! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
The manifest omits shared source dependencies, causing false source_unchanged decisions for JavaScript, onboarding consumers, Cypress, Playwright and WebdriverIO. Existing tests still execute, but the shadow fingerprints do not accurately represent their inputs.
🤖 Bits Code Review · Commit 99d437b
|
/datadog autotest review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aca729fffa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
The planner omits affected framework smokes for JavaScript and cross-framework shared helpers, allowing the required gate to succeed without those checks. Both ownership gaps were reproduced, and the localized inline replacements were verified.
🤖 Bits Code Review · Commit aca729f
|
/datadog autotest review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9bf86a0d2d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
JavaScript-only source changes also select Maven and Gradle smokes because Java ownership uses java* globs. This adds unrelated required jobs to the affected-check fast path.
🤖 Bits Code Review · Commit 9bf86a0
|
/datadog autotest review |
There was a problem hiding this comment.
More details
The aggregate gate rejects incomplete selected jobs, and missing base history selects all PR checks. Full framework workflows and dependency downloads were not exercised.
🤖 Bits Code Review · Commit ad32c63
There was a problem hiding this comment.
Changes to the shared execution helper in internal/testdrive/mocha.go select only Mocha and onboarding smokes. Java, Ruby, and Cucumber consumers lose their affected smoke coverage, allowing regressions through PR checks.
🤖 Bits Code Review · Commit ad32c63
|
/datadog autotest review |
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
More details
The required PR gate rejects incomplete selected jobs, and releases remain dependent on full compatibility. Full integration execution was not verified.
🤖 Bits Code Review · Commit 25c994b
25c994b to
abedd19
Compare
|
/datadog autotest review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
Changes to shared coverage helpers omit Python, Ruby, and Java smokes from PR checks, allowing the required gate to pass without exercising those consumers.
🤖 Bits Code Review · Commit abedd19
|
/datadog autotest review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5a830ae69c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
More details
Planner and workflow-focused checks support the selection and gating logic. External framework smokes and the full release compatibility matrix remain unverified.
🤖 Bits Code Review · Commit 5a830ae
|
/datadog autotest review @codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9540726f51
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
/datadog autotest review |
75c5969 to
d85e673
Compare
I've removed the build impact analysis pieces and kept the others. We'll try to merge this PR |
|
/datadog autotest review |
|
Codex Review: Didn't find any major issues. Swish! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
Local WebdriverIO integration runs can delete the shared CLI executable during per-test cleanup while its path remains cached, causing subsequent journeys to fail.
🤖 Bits Code Review · Commit a2ac82c
|
/datadog autotest review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
More details
The remaining uncertainty is end-to-end CI execution: the full matrix was not run, and its status could not be read because the query returned HTTP 403.
🤖 Bits Code Review · Commit 07d6df4
Isolate crash-recovery helpers and allocation measurements from CI reporting while retaining the original behavioral checks and memory limits. DD_1TLG1W DD_AYZRD6
07d6df4 to
fdf392e
Compare
|
/datadog autotest review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
More details
The shared CLI’s process-owned storage addresses the temporary-directory lifetime issue raised in discussion. Full GitHub Actions compatibility matrices were not executed.
🤖 Bits Code Review · Commit fdf392e
What
Why
Repeated CLI builds, dependency installation, test-binary compilation during shard discovery and SDK setup add work before useful test execution. Immutable dependency-only caches also fail to retain refreshed compiler output. Background instrumentation can interfere with helper protocols and process-wide allocation measurements. A shared local executable must also outlive the temporary directory of whichever journey builds it first. These changes reduce that overhead even when Test Impact Analysis chooses to run the tests.
CI keeps the repository's existing test policy. There is no additional changed-file selector or reduced PR-only onboard/testdrive suite.
E2E testing
Prerequisites: access to this branch's GitHub Actions runs and permission to rerun CI. For the local matrix inspection, install Node 24, npm and Python 3, and allow npm registry access. Also install the Go version in go.mod for the discovery harness. Use a disposable branch for the failure and cache-invalidation scenarios.
Open a CI run for this branch. Inspect the workflow graph and framework/version/shard combinations against main. Expect every existing onboard/testdrive journey and both Go instrumentation paths to remain. With the existing Datadog secret configured, verify that the Go jobs still configure Test Optimization and collect atomic coverage.
Inspect
test-binaryand a Jest, Java and browser integration job. Expect oneintegration-cliartifact from the current run, downloaded into each public CLI job and made executable. The journeys must useDDTEST_TEST_BINARYand retain their existing passing/failing-fixture and feature-control assertions.Rerun CI on the same commit. In matching Jest and Cucumber workers, inspect the locked-fixture key and restore/install steps. Expect an exact key match to skip
npm ci. A cold cache must runnpm ciand save the prepared dependencies before validation starts.On a disposable branch, change a pinned Jest dependency version in the matrix and push. Expect a different lockfile key and a clean install rather than reuse of the old fixture. Restore the version afterwards.
Inspect the Cucumber plan locally:
Expect all 13 requested lanes to be represented. If
latestresolves to13.3.0, the four equivalent pinned/latest shard pairs share executions, producing 9 jobs. A different latest version retains separate jobs. Compare aliases, exact Node/npm versions, test filters, shards and lockfiles; distinct execution inputs must stay separate.On the disposable branch, temporarily point a public CLI job's exported
DDTEST_TEST_BINARYat a nonexistent absolute path. Push and inspect that job. Expect a clear failure naming the invalid executable; it must not silently rebuild another binary. Restore the path.Review the Release workflow definition: its
cijob must call the sameci.ymlused by branches, and publication must depend on bothciand the existing Gocompatibilitygate. No release dispatch is needed for this inspection.Inspect the Go jobs' cache steps on a cold run and a rerun. Expect different namespaces for Orchestrion and
ddtest instrument, with OS, architecture, Go version and revision in their keys. The Orchestrion cache must include its executable. The configuration action must still run, set the current job's instrumentation environment and resolve compatible dependencies, while avoiding a second module-cache restore. On a new commit with unchanged dependencies, expect a compatible previous cache to restore and a fresh revision entry to be saved after success.Check discovery without executing a fixture's initialization:
Expect only
TestVisible, with no panic and a successful exit. Replace the fixture's package declaration with invalid syntax and repeat: expect a clear error and nonzero exit rather than a successful empty selection. In CI, inspect the Select Go test shard step and expect the existing deterministic shard filter without compiling all test binaries.In a Go job with Test Optimization enabled, inspect the service crash-recovery and Jenkins allocation-budget test entries. Expect the parent tests to remain reported. The service helper must not create an independent CI session; service readiness, termination and cleanup assertions remain. Jenkins must retain its 8 MiB and 16 MiB budgets and all parser behavior checks, with measurements isolated from background SDK allocations.
Verify local CLI reuse survives an individual journey's cleanup. From the repository root, create and run this temporary harness (the single benchmark iteration only supplies Go's testing interface; inspect its printed paths and executable behavior):
Expect
reused <executable> after deleting <journey>and a successful exit. The executable must be outside the removed journey directory, keep the same path on the second lookup, and successfully run--helpafter cleanup. The harness removes its process-owned binary storage when it exits.Cleanup: remove the local
qa_diranddiscovery_qadirectories and delete the disposable QA branch after restoring its temporary changes.