Skip to content

Reduce CI startup overhead and stabilize test helpers - #299

Merged
juan-fernandez merged 14 commits into
mainfrom
juan-fernandez/ci-selective-planner
Nov 5, 2026
Merged

juan-fernandez merged 14 commits into
mainfrom
juan-fernandez/ci-selective-planner

Conversation

@juan-fernandez

@juan-fernandez juan-fernandez commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What

  • Build one static CLI executable per CI run and share it across the existing public onboard/testdrive journeys. Local integration tests build it once in storage owned by TestMain, so the executable survives individual journeys changing or deleting their temporary directories.
  • Reuse prepared Jest and Cucumber dependencies when the complete lockfile, OS, architecture, Node and npm versions match. Save clean fixtures before tests can modify them.
  • Resolve Cucumber versions once and coalesce only identical runtime/dependency/scenario/shard combinations.
  • Disable Datadog reporting for Python prerequisite version checks; keep the actual validation processes and all six feature assertions.
  • Discover Go tests from build-selected source files instead of compiling every test executable; refresh compiler-specific caches per revision and reuse them during Test Optimization configuration.
  • Isolate crash-recovery helpers from CI reporting and Jenkins memory measurements from background allocations, retaining the original assertions and limits. Preserve Git stderr when the release fixture fails.
  • Run the same CI workflow for branch pushes and releases. Preserve all existing scenarios, compatibility matrices, TIA instrumentation, coverage and both Go instrumentation paths.

Why

Repeated CLI builds, dependency installation, test-binary compilation during shard discovery and SDK setup add work before useful test execution. Immutable dependency-only caches also fail to retain refreshed compiler output. Background instrumentation can interfere with helper protocols and process-wide allocation measurements. A shared local executable must also outlive the temporary directory of whichever journey builds it first. These changes reduce that overhead even when Test Impact Analysis chooses to run the tests.

CI keeps the repository's existing test policy. There is no additional changed-file selector or reduced PR-only onboard/testdrive suite.

E2E testing

Prerequisites: access to this branch's GitHub Actions runs and permission to rerun CI. For the local matrix inspection, install Node 24, npm and Python 3, and allow npm registry access. Also install the Go version in go.mod for the discovery harness. Use a disposable branch for the failure and cache-invalidation scenarios.

  1. Open a CI run for this branch. Inspect the workflow graph and framework/version/shard combinations against main. Expect every existing onboard/testdrive journey and both Go instrumentation paths to remain. With the existing Datadog secret configured, verify that the Go jobs still configure Test Optimization and collect atomic coverage.

  2. Inspect test-binary and a Jest, Java and browser integration job. Expect one integration-cli artifact from the current run, downloaded into each public CLI job and made executable. The journeys must use DDTEST_TEST_BINARY and retain their existing passing/failing-fixture and feature-control assertions.

  3. Rerun CI on the same commit. In matching Jest and Cucumber workers, inspect the locked-fixture key and restore/install steps. Expect an exact key match to skip npm ci. A cold cache must run npm ci and save the prepared dependencies before validation starts.

  4. On a disposable branch, change a pinned Jest dependency version in the matrix and push. Expect a different lockfile key and a clean install rather than reuse of the old fixture. Restore the version afterwards.

  5. Inspect the Cucumber plan locally:

    qa_dir=$(mktemp -d)
    python3 scripts/ci_cucumber_matrix.py --output "$qa_dir/fixtures" --summary "$qa_dir/summary.md"
    cat "$qa_dir/summary.md"
    cat "$qa_dir/fixtures/matrix.json"

    Expect all 13 requested lanes to be represented. If latest resolves to 13.3.0, the four equivalent pinned/latest shard pairs share executions, producing 9 jobs. A different latest version retains separate jobs. Compare aliases, exact Node/npm versions, test filters, shards and lockfiles; distinct execution inputs must stay separate.

  6. On the disposable branch, temporarily point a public CLI job's exported DDTEST_TEST_BINARY at a nonexistent absolute path. Push and inspect that job. Expect a clear failure naming the invalid executable; it must not silently rebuild another binary. Restore the path.

  7. Review the Release workflow definition: its ci job must call the same ci.yml used by branches, and publication must depend on both ci and the existing Go compatibility gate. No release dispatch is needed for this inspection.

  8. Inspect the Go jobs' cache steps on a cold run and a rerun. Expect different namespaces for Orchestrion and ddtest instrument, with OS, architecture, Go version and revision in their keys. The Orchestrion cache must include its executable. The configuration action must still run, set the current job's instrumentation environment and resolve compatible dependencies, while avoiding a second module-cache restore. On a new commit with unchanged dependencies, expect a compatible previous cache to restore and a fresh revision entry to be saved after success.

  9. Check discovery without executing a fixture's initialization:

    discovery_qa=$(mktemp -d)
    go build -o "$discovery_qa/testnames" ./scripts/testnames
    mkdir "$discovery_qa/fixture"
    printf 'module example.com/discovery\n\ngo 1.24\n' > "$discovery_qa/fixture/go.mod"
    cat > "$discovery_qa/fixture/fixture_test.go" <<'EOF'
    package fixture
    import "testing"
    func init() { panic("discovery executed the test binary") }
    func TestMain(m *testing.M) { panic("discovery executed TestMain") }
    func TestVisible(t *testing.T) {}
    EOF
    (cd "$discovery_qa/fixture" && "$discovery_qa/testnames")

    Expect only TestVisible, with no panic and a successful exit. Replace the fixture's package declaration with invalid syntax and repeat: expect a clear error and nonzero exit rather than a successful empty selection. In CI, inspect the Select Go test shard step and expect the existing deterministic shard filter without compiling all test binaries.

  10. In a Go job with Test Optimization enabled, inspect the service crash-recovery and Jenkins allocation-budget test entries. Expect the parent tests to remain reported. The service helper must not create an independent CI session; service readiness, termination and cleanup assertions remain. Jenkins must retain its 8 MiB and 16 MiB budgets and all parser behavior checks, with measurements isolated from background SDK allocations.

  11. Verify local CLI reuse survives an individual journey's cleanup. From the repository root, create and run this temporary harness (the single benchmark iteration only supplies Go's testing interface; inspect its printed paths and executable behavior):

qa_harness=$(mktemp -d "$PWD/.qa-cli-lifetime.XXXXXX")
cat > "$qa_harness/main.go" <<'EOF'
package main
import (
 "flag"
 "fmt"
 "os"
 "os/exec"
 "testing"
 "github.com/DataDog/ddtest/internal/testutil"
)
func main() {
 testing.Init()
 if err := flag.Set("test.benchtime", "1x"); err != nil { panic(err) }
 root, err := os.MkdirTemp("", "ddtest-process-qa-")
 if err != nil { panic(err) }
 defer os.RemoveAll(root)
 testutil.SetBinaryDirectory(root)
 os.Unsetenv("DDTEST_TEST_BINARY")
 result := testing.Benchmark(func(b *testing.B) {
  journey, err := os.MkdirTemp(root, "journey-")
  if err != nil { b.Fatal(err) }
  os.Setenv("TMPDIR", journey)
  first := testutil.DDTest(b)
  if err := os.RemoveAll(journey); err != nil { b.Fatal(err) }
  os.Setenv("TMPDIR", root)
  second := testutil.DDTest(b)
  if first != second { b.Fatal("executable was not reused") }
  if output, err := exec.Command(second, "--help").CombinedOutput(); err != nil {
   b.Fatalf("cached executable failed: %v\n%s", err, output)
  }
  fmt.Printf("reused %s after deleting %s\n", second, journey)
 })
 if result.N != 1 { os.Exit(1) }
}
EOF
(cd internal/testdrive && go run "$qa_harness/main.go")
rm -rf "$qa_harness"

Expect reused <executable> after deleting <journey> and a successful exit. The executable must be outside the removed journey directory, keep the same path on the second lookup, and successfully run --help after cleanup. The harness removes its process-owned binary storage when it exits.

Cleanup: remove the local qa_dir and discovery_qa directories and delete the disposable QA branch after restoring its temporary changes.

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Pipelines  Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 80.85%
• Overall Coverage: 88.52% (-0.16%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fdf392e | Docs | View more details | Give us feedback!

@juan-fernandez

Copy link
Copy Markdown
Contributor Author

/datadog autotest review
@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-05T19:26:17.896994Z fdf392e Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 99d437bfaa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The manifest omits shared source dependencies, causing false source_unchanged decisions for JavaScript, onboarding consumers, Cypress, Playwright and WebdriverIO. Existing tests still execute, but the shadow fingerprints do not accurately represent their inputs.

Open Bits AI session

🤖 Bits Code Review · Commit 99d437b

Comment thread scripts/ci_test_plan.json Resolved
Comment thread scripts/ci_test_plan.json Resolved
Comment thread scripts/ci_test_plan.json Resolved
Comment thread scripts/ci_test_plan.json Resolved
Comment thread scripts/ci_test_plan.json Resolved
@juan-fernandez juan-fernandez changed the title Add shadow planning for selective integration tests Run affected PR checks and gate releases on full compatibility Nov 3, 2026
@juan-fernandez

Copy link
Copy Markdown
Contributor Author

/datadog autotest review
@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aca729fffa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/ci.yml Resolved

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The planner omits affected framework smokes for JavaScript and cross-framework shared helpers, allowing the required gate to succeed without those checks. Both ownership gaps were reproduced, and the localized inline replacements were verified.

Open Bits AI session

🤖 Bits Code Review · Commit aca729f

Comment thread scripts/ci_test_plan.json Resolved
Comment thread scripts/ci_test_plan.json Resolved
@juan-fernandez

Copy link
Copy Markdown
Contributor Author

/datadog autotest review
@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9bf86a0d2d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/ci_test_plan.py Resolved

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

JavaScript-only source changes also select Maven and Gradle smokes because Java ownership uses java* globs. This adds unrelated required jobs to the affected-check fast path.

Open Bits AI session

🤖 Bits Code Review · Commit 9bf86a0

Comment thread scripts/ci_test_plan.json Resolved
@juan-fernandez

Copy link
Copy Markdown
Contributor Author

/datadog autotest review
@codex review

@juan-fernandez
juan-fernandez marked this pull request as ready for review November 4, 2026 10:42
@juan-fernandez
juan-fernandez requested a review from a team as a code owner November 4, 2026 10:42

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The aggregate gate rejects incomplete selected jobs, and missing base history selects all PR checks. Full framework workflows and dependency downloads were not exercised.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit ad32c63

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Changes to the shared execution helper in internal/testdrive/mocha.go select only Mocha and onboarding smokes. Java, Ruby, and Cucumber consumers lose their affected smoke coverage, allowing regressions through PR checks.

Open Bits AI session

🤖 Bits Code Review · Commit ad32c63

Comment thread scripts/ci_test_plan.json Resolved
@juan-fernandez

Copy link
Copy Markdown
Contributor Author

/datadog autotest review
@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 25c994beaa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The required PR gate rejects incomplete selected jobs, and releases remain dependent on full compatibility. Full integration execution was not verified.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 25c994b

@juan-fernandez
juan-fernandez force-pushed the juan-fernandez/ci-selective-planner branch from 25c994b to abedd19 Compare November 4, 2026 20:15
@juan-fernandez

Copy link
Copy Markdown
Contributor Author

/datadog autotest review
@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: abedd19253

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Changes to shared coverage helpers omit Python, Ruby, and Java smokes from PR checks, allowing the required gate to pass without exercising those consumers.

Open Bits AI session

🤖 Bits Code Review · Commit abedd19

Comment thread scripts/ci_test_plan.json Resolved
@juan-fernandez

Copy link
Copy Markdown
Contributor Author

/datadog autotest review
@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5a830ae69c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/ci_smoke.py Resolved

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Planner and workflow-focused checks support the selection and gating logic. External framework smokes and the full release compatibility matrix remain unverified.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 5a830ae

Copy link
Copy Markdown
Contributor Author

/datadog autotest review @codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9540726f51

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/ci_test_plan.py Resolved
@juan-fernandez

Copy link
Copy Markdown
Contributor Author

/datadog autotest review
@codex review

@juan-fernandez
juan-fernandez force-pushed the juan-fernandez/ci-selective-planner branch from 75c5969 to d85e673 Compare November 5, 2026 14:14
@juan-fernandez

Copy link
Copy Markdown
Contributor Author

we'll hold this off to see how TIA behaves in the repo

I've removed the build impact analysis pieces and kept the others. We'll try to merge this PR

@juan-fernandez juan-fernandez changed the title Reuse CI builds and locked dependency fixtures Reduce CI startup overhead and stabilize test helpers Nov 5, 2026
@juan-fernandez

Copy link
Copy Markdown
Contributor Author

/datadog autotest review
@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: a2ac82c503

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Local WebdriverIO integration runs can delete the shared CLI executable during per-test cleanup while its path remains cached, causing subsequent journeys to fail.

Open Bits AI session

🤖 Bits Code Review · Commit a2ac82c

Comment thread internal/testutil/binary.go Resolved
@juan-fernandez

Copy link
Copy Markdown
Contributor Author

/datadog autotest review
@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep them coming!

Reviewed commit: 07d6df4424

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The remaining uncertainty is end-to-end CI execution: the full matrix was not run, and its status could not be read because the query returned HTTP 403.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 07d6df4

@juan-fernandez
juan-fernandez force-pushed the juan-fernandez/ci-selective-planner branch from 07d6df4 to fdf392e Compare November 5, 2026 18:25
@juan-fernandez

Copy link
Copy Markdown
Contributor Author

/datadog autotest review
@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: fdf392efda

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The shared CLI’s process-owned storage addresses the temporary-directory lifetime issue raised in discussion. Full GitHub Actions compatibility matrices were not executed.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit fdf392e

@juan-fernandez
juan-fernandez merged commit 79998cb into main Nov 5, 2026
129 checks passed
@juan-fernandez
juan-fernandez deleted the juan-fernandez/ci-selective-planner branch November 5, 2026 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants