Skip to content

go: instrument Go tests with ddtest instead of Orchestrion - #290

Draft
tonyredondo wants to merge 18 commits into
mainfrom
tonyredondo/ddtest-instrument
Draft

tonyredondo wants to merge 18 commits into
mainfrom
tonyredondo/ddtest-instrument

Conversation

@tonyredondo

@tonyredondo tonyredondo commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Summary

The next major version of the Go support: Go tests instrumented by ddtest instrument bazel instead of Orchestrion, on upstream rules_go.

  • The instrumented Go SDK is the official SDK, or the workspace's own, with the ddtest executable as its compiler. Only instrumented tests select it, through a transition on @io_bazel_rules_go//go/toolchain:sdk_name; other builds fetch nothing.
  • Mini, ddtest's runtime, comes from a repository that ddtest writes, and each test gets its registration file. A test that would link a second copy of Mini fails at analysis.
  • ddtest is pinned to pre-release-f5df4db (DataDog/ddtest f5df4db), with the checksums of its executables.
  • Removed: the vendored rules_go forks and their tooling, Orchestrion, the dd-trace-go pins and pin files, orchestrion_mode and orchestrion_pin_files (passing them fails with a migration message).
  • Bzlmod: topt_go_ddtest.sdk(go_mod = ... | go_version = ...). WORKSPACE: upstream rules_go, then topt_go_ddtest_workspace_repositories(...). dd_topt_go_bootstrap writes and migrates both setups.
  • The instrumented toolchains honor @io_bazel_rules_go//go/toolchain:sdk_version like rules_go's own. More sdk tags with sdk_version (or sdk_variants in WORKSPACE) add SDK variants, such as a FIPS SDK for a go_cross_binary in a test's data; an sdk_version that no SDK serves fails with what to configure.

One commit per step. The Orchestrion path kept working until the removal commits, so the payloads below were compared on the same targets. The removal commits (from 3d69889) are split by area and are not green one by one; the PR is meant to be squash-merged.

Step Status
Refer to rules_go as io_bazel_rules_go Done
ddtest SDK and Mini repositories, Bzlmod extension, toolchains Done
Test rule, transition, registration, single-Mini check Done
dd_topt_go_test, metadata, stdlib warmup Done
WORKSPACE Done
CI and E2E, payload comparison Done
Removals: Starlark, forks and tooling, fixtures and CI, examples, onboarding Done
Docs, skills, changelog Done
Auditor Done (3 rounds, below)
sdk_version and SDK variants, from a review against a large consumer Done (auditor rounds below)
Review Next

Validation

  • cd modules/go && ../../bazelw test //... --override_module=datadog-rules-test-optimization=../..: 160 tests pass. run_go_helper_compatibility.sh passes and now also requires the Mini alias of configurable deps to be incompatible when its test is.
  • ./bazelw test //tools/... //examples/...: 213 tests; 212 passed in the full run and python_tools_test timed out in its pwsh logging subtests under that load, then passed alone. CI runs the same suite.
  • modules/go/tests/ddtest: instrumented_test runs instrumented and fails unless Mini set its context's trace (a transition that does not select the SDK makes it fail); foreign_mini_failure_test expects the analysis error for a foreign copy of Mini; the stdlib warmup builds with the instrumented SDK.
  • run_bzlmod_go_integration.sh and run_workspace_go_integration.sh, locally with FORBID_HOST_GO=1 (ddtest needs no host Go), at both ends of CI's matrix: rules_go 0.60.0 with Go 1.25.0 and rules_go 0.65.0 with Go 1.26.3, in both modes: pass. They check the bazel.go.instrumentation.* and bazel.go.sdk.* metadata; payloads, BEP freshness and the uploader dry run with --validate-enrichment; and, in the hermetic lane, with assert_ddtest_aquery.py in the default, opt and no-strip modes, that the test's compiles, link and standard library use the instrumented SDK, with the registration file and Mini's testopt. Bzlmod also checks that building the library alone fetches neither the SDK nor Mini, and contacts no metadata server with Test Optimization disabled. WORKSPACE instruments the registered @go_sdk through base_sdk_roots.
  • run_mock_server_tests.sh: passes (about 11 minutes), including the bootstrap scenarios (new managed block, migration from the Orchestrion block) and a guided test built with the instrumented SDK.
  • Repositories, by hand: without an sdk tag only topt_go_ddtest_toolchains is fetched; with one, only the SDK of the exec platform in use.

sdk_version and SDK variants

A read-only review against a large consumer found that a go_cross_binary(sdk_version = "<version>fips") in an instrumented test's data built with the default instrumented SDK: the transition sets sdk_name to ddto for everything below the test, and the toolchains matched only that. The Orchestrion fork turned the instrumentation off on tool and data edges, which upstream rules_go cannot do without patches.

  • Toolchains match sdk_version as rules_go's declare_bazel_toolchains does: the default SDK serves an unset value and the values of its go_version; a variant serves its own sdk_version. Variants have their own go_version, base_sdk_roots, exec_platforms and experiments.
  • A stand-in toolchain catches only an sdk_version that no SDK serves (an alias whose select() negates a config_setting_group of every served value) and fails with the requested value and the configured SDKs. It never matches a served value, so it cannot make an exec platform without an SDK eligible.
  • A base SDK whose VERSION differs from its go_version fails at fetch time; go_wrap_sdk also gets the version, for SDKs without a VERSION file.

Validation, on Linux amd64 (0d8394f): modules/go //... 163 tests, both integration scripts with FORBID_HOST_GO=1 at rules_go 0.60.0 and Go 1.25.0, and //tools/... //examples/... (213 tests; python_tools_test timed out in its pwsh subtests under load and passed alone) all pass. In particular:

  • modules/go/tests/ddtest:sdk_variants_test runs, from its data: a plain go_binary (go1.26.3, the default SDK), a go_cross_binary with sdk_version = "1.26" (go1.26.3), one with sdk_version = "1.25.9fips", a variant on Go 1.25.9 with boringcrypto (go1.25.9 X:boringcrypto), and the output of an exec tool that imports testing. With the toolchains' sdk_version setting removed, it fails: the FIPS binary prints go1.26.3.
  • Unit tests: matching (including 1.26rc2), variant validation, the generated toolchains and stand-in, the base SDK check.
  • run_bzlmod_go_integration.sh: a test whose data asks for an unconfigured sdk_version fails to build with the stand-in's message. run_workspace_go_integration.sh: an sdk_variants entry serves a go_cross_binary in the test's data, and a variant whose base SDK is not its go_version fails with that message.
  • By hand, in a copy of modules/go with both SDKs on linux_amd64 only and a fake linux_arm64 exec platform registered first: the build skips the fake platform and runs on the host; with the first version of the stand-in (no sdk_version condition), analysis picked the fake platform and failed.
Auditor finding (GPT-6.1-Sol, xhigh, 3 rounds, last one SHIP) Outcome
The stand-in, with no exec or target constraints, made exec platforms without an SDK eligible Fixed: it matches only unserved sdk_version values; reproduced and checked by hand as above
A base SDK could contradict the go_version that its toolchains advertise Fixed: checked against VERSION, and through go_wrap_sdk without one
Prerelease versions lost rules_go's major.minor selectors Fixed: rules_go's forms, unit case for 1.26rc2
Troubleshooting promised the stand-in's message for uncovered platforms Fixed: those get Bazel's own toolchain error, documented

Auditor (GPT-6.1-Sol, xhigh, 3 rounds)

Finding Outcome
Registration skipped sources with a negated build constraint (//go:build !windows) and failed with "no Go sources" Fixed: a file is skipped only when its constraint holds on no platform (ignore); unconstrained sources decide the package. Unit cases and constrained_test
A test whose sources all come through embed had no registration package Fixed: the registration also reads the embedded libraries' GoInfo.srcs. embed_only_test
deps as a select() that already names Mini's testopt got a duplicated label Fixed: Mini joins a select() through a private alias (same archive, one testopt.a in the link) that carries the test's target_compatible_with. select_deps_test and the helper compatibility fixture
A custom rules_go_repo_name in WORKSPACE could not work with the generated SDK repositories Fixed: the helper requires io_bazel_rules_go and says why
Versions still 1.4.0 Withdrawn: releases bump versions in their own PR
Every source constrained and an inactive one declaring another package (e.g. a //go:build tools file of package main in the test's srcs) Not fixed: gazelle does not produce it, the failure is loud ("sources declare packages ..."), and a full fix needs rules_go's private build context

Known limitations

  • Inside an instrumented test, every Go target, exec tools included, builds with an instrumented SDK (the default one or the variant its sdk_version selects). Its hooks do nothing in programs that do not link Mini. Building tools with the ordinary SDK would need transitions that upstream rules_go does not offer.
  • Not verified yet: remote execution, a real exec platform other than the host (only the fake one above, by hand), and nogo. The cross-repository E2E suites run ddtest in their own PR, green at 1428be7 on Linux, macOS and Windows; it is being repinned to this head.

Payloads compared with Orchestrion

The Bzlmod consumer fixture, built once with Orchestrion (test_optimization mode, dd-trace-go v2.9.1) and once with ddtest, on the same targets and mock backend: hello_test and a compare_test with passing, failing and skipped tests, subtests, parallel tests, examples and an external test package. Events are matched by type, module, suite and name; identifiers, timestamps, durations, runtime-id and process_id are ignored. The harness is a local copy of the fixture and stays out of the repository.

Settings Orchestrion events ddtest events Differences
Known tests and test management (the fixture's default) 23 + 8 25 + 8 ddtest also reports ExampleGreeting (pass) and ExampleWrong (fail); Orchestrion reports no examples
Same, plus auto test retries and early flake detection 58 + 8 60 + 8 Only the two examples. Retries match test by test: 5 auto_test_retry runs of TestFail, TestSubtests and each of its subtests, with the same test.final_status values. Early flake detection retried no test in either run, so it is not compared

Tags on every event:

Tag Orchestrion ddtest
language on each event and in the payload metadata in the payload metadata only
library_name (payload metadata) absent ddtest
_dd.library_capabilities.coverage_report_upload absent 1
_dd.tags.process, _dd.profiling.enabled, _dd.top_level, _dd.trace_span_attribute_schema, _sampling_priority_v1 present (dd-trace-go's APM tags) absent
error.stack dd-trace-go frames, testing/testing.go Mini frames under external/...topt_go_ddtest_mini, GOROOT/src/testing/testing.go

test.working_directory differs only because each build ran from its own workspace directory, so the execroots differ. Every other tag and value, statuses and source lines included, is the same. The test logs differ only by dd-trace-go's loading features: ... localhost:8126 error line, which ddtest does not print.

Release Readiness

  • Updated user-facing docs for behavior or API changes.
  • Updated LICENSE-3rdparty.csv when dependencies or vendored code changed.
  • Confirmed no secrets, credentials, or internal-only details were added.
  • Confirmed security-sensitive behavior was tested or explicitly marked not applicable.

The SDK repositories that rules_go's go_wrap_sdk writes load rules_go as
@io_bazel_rules_go, and rules_go's SDK macros name it so in strings that
resolve in those repositories. The Go companion will create such repositories
for the ddtest-instrumented SDK, so it now sees rules_go under that name, and
so do the root module, which loads the companion as a local repository, and
the examples.

The companion's WORKSPACE repo_mapping, the release INSTALL generator and the
rules_go profile smoke follow. No behavior changes.
topt_go_ddtest_repository.bzl declares, from one pinned ddtest pre-release
(pre-release-f5df4db, with each executable's sha256):
- the repository host's ddtest executable;
- Mini's repository, which that executable writes with
  `ddtest instrument bazel --mini`;
- for each exec platform, a base Go SDK, the official one from go_version or
  a root label of the workspace's own, a copy of it whose compiler is that
  platform's ddtest executable (`ddtest instrument bazel --goroot`), and the
  go_wrap_sdk repository around it, with the base SDK's experiments;
- one toolchain per exec and target platform, gated on
  @io_bazel_rules_go//go/toolchain:sdk_name == "ddto".
Every output's DDTEST_INSTRUMENT is checked against the pinned contract,
sdk_name and runtime version, which is the release tag with v0.0.0- before it.

The topt_go_ddtest extension reads the root module's sdk tag: go_mod (the
toolchain directive, then the go directive, as go_sdk.from_file), go_version,
base_sdk_roots per exec platform, exec_platforms, experiments and panic_hook.
The Go companion registers the toolchains, so the root module registers
none. Without an sdk tag, a single toolchain stands in that fails, when an
instrumented test selects it, with what to configure; other builds fetch
nothing, as a test of the companion checks.

The companion's own tests configure Go 1.26.3. Checked by hand: Mini's
repository builds, the linux_amd64 SDK's compiler is ddtest with compile.real
next to it, and with sdk_name=ddto the standard library builds with it, while
without it the default SDK is used.
dd_topt_go_test(instrumentation = "ddtest") builds the hidden raw go_test
with the SDK that `ddtest instrument bazel` instruments. The Orchestrion path
stays the default until it is removed.

- topt_go_ddtest.bzl: the public wrapper, with the transition that sets
  @io_bazel_rules_go//go/toolchain:sdk_name to "ddto" and the same launcher,
  metadata copy and execution policy as the Orchestrion wrapper; the raw test
  keeps the caller's data, so $(rlocationpath) in x_defs still expands.
- The raw test gets Mini's testopt, unless its deps already name it, as they
  do after a workspace overrides go_deps' copy of Mini with the generated one.
- It gets the registration file that ddtest instrument writes, from a small
  Go tool that reads the sources' package clauses: the external test package
  when a source declares one, the package under test otherwise. srcs may be a
  select, a filegroup or generated.
- An aspect over deps and embed stops analysis when a package of
  github.com/DataDog/ddtest/ddto comes from a repository other than the
  generated Mini: rules_go 0.64 links one copy of each package without an
  error. The message shows the use_repo and override_repo recipe.
- Target metadata reports bazel.go.instrumentation.backend, version
  (v0.0.0-pre-release-f5df4db), release and contract, bazel.go.sdk.name and
  the selected SDK's version, instead of bazel.go.orchestrion.*.
- No Orchestrion pin files. Like Orchestrion's test_optimization mode, the
  test binary linker optimization applies and no-remote-exec stays with the
  TestRunner.

tests/ddtest: a test that fails unless Mini set its context's trace, with an
external test package and testopt in its deps, and an analysis test for a
foreign copy of Mini. A transition that does not select the instrumented SDK
fails the first, and removing the testopt dedupe fails its analysis.
dd_topt_go_stdlib_warmup(instrumentation = "ddtest") builds the standard
library with the same transitions as a ddtest-instrumented go_test, so with
the SDK that `ddtest instrument bazel` instruments. The Orchestrion warmup
stays the default.

tests/ddtest: the warmup target has outputs, and the transition sets
sdk_name to "ddto". Checked with aquery: the ddtest warmup's GoStdlib action
uses the instrumented linux_amd64 SDK, the Orchestrion one the default SDK.
topt_go_ddtest_workspace_repositories declares the same repositories as
the Bzlmod extension and registers their toolchains, which the Go
companion module does by itself under Bzlmod.

Checked by hand in a WORKSPACE consumer outside the repository: with
the rules_go fork that the Orchestrion loads still need, an instrumented
dd_topt_go_test passes, its standard library comes from the instrumented
SDK, and only the linux_amd64 SDK is fetched; base_sdk_roots pointing at
the registered @go_sdk works too.
INSTRUMENTATION=ddtest runs the positive WORKSPACE and Bzlmod fixtures
with the test instrumented by ddtest instrument instead of Orchestrion:

- Bzlmod declares topt_go_ddtest.sdk(go_version = ...), and WORKSPACE
  calls topt_go_ddtest_workspace_repositories with the registered
  @go_sdk as the base SDK of the host's exec platform.
- The test checks the bazel.go.instrumentation.* and bazel.go.sdk.*
  metadata and that no Orchestrion metadata is left.
- The hermetic lane checks with assert_ddtest_aquery.py, in the default,
  opt and no-strip modes, that the test's compiles, link and standard
  library use the instrumented SDK, that the compile gets the
  registration file and the link Mini's testopt library, and that only
  the TestRunner has no-remote-exec.
- Payload, BEP freshness and uploader dry-run checks are the same as
  Orchestrion's.

The Orchestrion-only scenarios (pin files, the tool repository's
errors, the disabled and Windows smokes) stay Orchestrion's. CI runs the
ddtest fixtures in every rules_go compatibility shard, without the
host's Go.
…xture

ddtest_host_platform did not know Git Bash's MINGW/MSYS uname, so the
Python test that writes the ddtest WORKSPACE failed on Windows.
dd_topt_go_test builds every enabled test with the SDK that ddtest
instruments: the instrumentation, orchestrion_mode and
orchestrion_pin_files parameters are gone, and passing orchestrion_mode
or orchestrion_pin_files fails with a migration message instead of
reaching go_test. The test-binary linker flags apply by default, and
no-remote-exec always stays on the TestRunner only.

The target metadata always carries bazel.go.instrumentation.* and
bazel.go.sdk.*, never bazel.go.orchestrion.*. dd_topt_go_stdlib_warmup
always warms the instrumented standard library.

topt_go_orchestrion.bzl and topt_go_orchestrion_repository.bzl are
removed; the launcher tests now cover ddtest_go_test, and the tests that
only covered Orchestrion's modes, pin files and transition are removed.
The Go companion and the root module use upstream rules_go from the
BCR. Removed: third_party/rgo and third_party/rules_go_orchestrion, the
fork registry, materializer, diff, profile, smoke and drift tooling,
their tests and CI jobs, and the release archive's fork checks.

The WORKSPACE helper datadog_go_test_optimization_workspace_repositories
now declares only the Go companion, like the Python and Java helpers:
the consumer declares upstream rules_go first, and rules_go_repo_name
only maps it. Release notes and release.json drop the fork blocks.
The WORKSPACE and Bzlmod fixtures now only cover ddtest instrument, on
upstream rules_go (RULES_GO_VERSION: a BCR bazel_dep, or the release zip
with its BCR integrity in WORKSPACE). The Bzlmod fixture also checks
that a build of the library alone fetches neither the instrumented SDK
nor Mini, and contacts no metadata server while Test Optimization is
disabled. WINDOWS_SMOKE_ONLY runs the instrumented test on Windows.

The Orchestrion-only scenarios (pin files, the tool repository and its
errors, the disabled and config-transition smokes, measure mode) and
their helpers are removed.

CI: workspace-compat runs rules_go 0.60.0, 0.61.1, 0.62.0, 0.63.0,
0.64.2 and 0.65.0 (with Go 1.26) in both modes without the host's Go;
windows-go-smoke replaces the Orchestrion Windows bootstrap lane, the
no-host-Go Orchestrion lane goes, and the Go examples build on Windows
again.
The example modules configure topt_go_ddtest from the Go module's
go.mod instead of pointing at the Orchestrion bootstrap, and drop the
Orchestrion flag, orchestrion.tool.go, orchestrion.yml and the
dd-trace-go and Orchestrion requirements that only served as pins.
dd_topt_go_bootstrap writes a managed MODULE.bazel block that
configures topt_go_ddtest.sdk (go_version from --runtime-version, or
go_mod from the Go module's go.mod) instead of the rules_go fork
override and the orchestrion extension. Rerunning it replaces the old
Orchestrion block, and it stops with a message when an unmanaged
rules_go override still points at the removed fork. The WORKSPACE
snippet declares upstream rules_go first and calls
topt_go_ddtest_workspace_repositories; wrappers no longer pass
Orchestrion attributes.

Removed with Orchestrion: the dd-trace-go pinning and go.mod sync, the
Orchestrion files, the go_repository drift check, the fork and archive
overrides and their flags. The onboarding pins keep only the
rules_test_optimization commit and archive.

The mock-server suite's multi-service scenario uses upstream rules_go
and checks the hidden raw test, and its bootstrap scenarios cover the
new block, the migration from the old one and a guided instrumented
test.
The message pointed WORKSPACE users at topt_go_ddtest_repositories, which
registers no toolchains; topt_go_ddtest_workspace_repositories does. The
macro's comments name Mini instead of the tracer.
README, CONTRIBUTING, AGENTS and the references describe the Bzlmod and
WORKSPACE setup with topt_go_ddtest, upstream rules_go, the bootstrap's
managed block, the target metadata, the integration fixtures and CI
lanes, and the ddtest failure modes in Troubleshooting. The Go
onboarding skill is rewritten for ddtest; the large-monorepo rollout
guide keeps its generic checklist.

Removed: the Orchestrion and rules_go fork documents and the
rules-go-orchestrion-upstream-migration skill. CHANGELOG's Unreleased
section lists the changes and how Go consumers migrate.
The registration tool treated every build tag but ignore as set, so a
test whose sources all carry a negated constraint such as !windows had
no package and failed with "no Go sources". A file is now skipped only
when its constraint holds on no platform, as with ignore. The
registration rule also reads the sources of the libraries the test
embeds, as rules_go compiles them into the test.

The Go WORKSPACE helper rejects a rules_go_repo_name other than
io_bazel_rules_go: the SDK repositories that ddtest instrument writes
load rules_go under that name. Troubleshooting covers a select() in
deps that also lists Mini's testopt.
Sources without a build constraint now decide the registration
package; constrained ones only decide when every source has one, so a
file that the configuration excludes, such as a //go:build tools file
of another package, no longer conflicts. When deps is a select(), which
the macro cannot inspect, Mini joins it through a private alias: the
same archive under another label, so a select that already names
testopt no longer duplicates it.
The alias that adds Mini to configurable deps did not carry the test's
target_compatible_with, so a wildcard cquery over an incompatible test
analyzed it and fetched Mini. It now has the test's constraints, and the
helper compatibility fixture and its cquery check cover it.

The foreign-Mini analysis error also tells WORKSPACE users what to do,
and Troubleshooting describes that instrumented tests ignore an
sdk_version transition below them.
Inside an instrumented test every Go target resolves to an instrumented
SDK, because the test's transition sets sdk_name to ddto and the
toolchains matched only that. A go_cross_binary(sdk_version = ...) below
the test, such as a FIPS binary in data, then built silently with the
default instrumented SDK.

The toolchains now match sdk_version as rules_go matches its own: the
default SDK serves an unset value and the values of its go_version
(major, major.minor, major.minor.patch and prerelease forms). More
topt_go_ddtest.sdk tags with sdk_version, or sdk_variants in WORKSPACE,
add SDK variants with their own go_version, base_sdk_roots,
exec_platforms and experiments. A stand-in toolchain catches only an
sdk_version that no SDK serves and fails with what to configure; it
never matches a served value, so it cannot make an exec platform without
an SDK eligible. A base SDK whose VERSION is not its go_version now
fails with a message that says so.

Tests: unit tests for the matching, variant validation, generated
toolchains and base SDK check; sdk_variants_test runs data binaries
built with the default SDK, a version prefix and a boringcrypto variant
on another Go version, and an exec tool that imports testing; the Bzlmod
integration checks the unconfigured-sdk_version failure and the WORKSPACE
integration a variant and the stale base SDK failure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant