Repository navigation
go: instrument Go tests with ddtest instead of Orchestrion - #290
Draft
tonyredondo wants to merge 18 commits into
Draft
tonyredondo wants to merge 18 commits into
tonyredondo wants to merge 18 commits into
Conversation
The SDK repositories that rules_go's go_wrap_sdk writes load rules_go as @io_bazel_rules_go, and rules_go's SDK macros name it so in strings that resolve in those repositories. The Go companion will create such repositories for the ddtest-instrumented SDK, so it now sees rules_go under that name, and so do the root module, which loads the companion as a local repository, and the examples. The companion's WORKSPACE repo_mapping, the release INSTALL generator and the rules_go profile smoke follow. No behavior changes.
topt_go_ddtest_repository.bzl declares, from one pinned ddtest pre-release (pre-release-f5df4db, with each executable's sha256): - the repository host's ddtest executable; - Mini's repository, which that executable writes with `ddtest instrument bazel --mini`; - for each exec platform, a base Go SDK, the official one from go_version or a root label of the workspace's own, a copy of it whose compiler is that platform's ddtest executable (`ddtest instrument bazel --goroot`), and the go_wrap_sdk repository around it, with the base SDK's experiments; - one toolchain per exec and target platform, gated on @io_bazel_rules_go//go/toolchain:sdk_name == "ddto". Every output's DDTEST_INSTRUMENT is checked against the pinned contract, sdk_name and runtime version, which is the release tag with v0.0.0- before it. The topt_go_ddtest extension reads the root module's sdk tag: go_mod (the toolchain directive, then the go directive, as go_sdk.from_file), go_version, base_sdk_roots per exec platform, exec_platforms, experiments and panic_hook. The Go companion registers the toolchains, so the root module registers none. Without an sdk tag, a single toolchain stands in that fails, when an instrumented test selects it, with what to configure; other builds fetch nothing, as a test of the companion checks. The companion's own tests configure Go 1.26.3. Checked by hand: Mini's repository builds, the linux_amd64 SDK's compiler is ddtest with compile.real next to it, and with sdk_name=ddto the standard library builds with it, while without it the default SDK is used.
dd_topt_go_test(instrumentation = "ddtest") builds the hidden raw go_test with the SDK that `ddtest instrument bazel` instruments. The Orchestrion path stays the default until it is removed. - topt_go_ddtest.bzl: the public wrapper, with the transition that sets @io_bazel_rules_go//go/toolchain:sdk_name to "ddto" and the same launcher, metadata copy and execution policy as the Orchestrion wrapper; the raw test keeps the caller's data, so $(rlocationpath) in x_defs still expands. - The raw test gets Mini's testopt, unless its deps already name it, as they do after a workspace overrides go_deps' copy of Mini with the generated one. - It gets the registration file that ddtest instrument writes, from a small Go tool that reads the sources' package clauses: the external test package when a source declares one, the package under test otherwise. srcs may be a select, a filegroup or generated. - An aspect over deps and embed stops analysis when a package of github.com/DataDog/ddtest/ddto comes from a repository other than the generated Mini: rules_go 0.64 links one copy of each package without an error. The message shows the use_repo and override_repo recipe. - Target metadata reports bazel.go.instrumentation.backend, version (v0.0.0-pre-release-f5df4db), release and contract, bazel.go.sdk.name and the selected SDK's version, instead of bazel.go.orchestrion.*. - No Orchestrion pin files. Like Orchestrion's test_optimization mode, the test binary linker optimization applies and no-remote-exec stays with the TestRunner. tests/ddtest: a test that fails unless Mini set its context's trace, with an external test package and testopt in its deps, and an analysis test for a foreign copy of Mini. A transition that does not select the instrumented SDK fails the first, and removing the testopt dedupe fails its analysis.
dd_topt_go_stdlib_warmup(instrumentation = "ddtest") builds the standard library with the same transitions as a ddtest-instrumented go_test, so with the SDK that `ddtest instrument bazel` instruments. The Orchestrion warmup stays the default. tests/ddtest: the warmup target has outputs, and the transition sets sdk_name to "ddto". Checked with aquery: the ddtest warmup's GoStdlib action uses the instrumented linux_amd64 SDK, the Orchestrion one the default SDK.
topt_go_ddtest_workspace_repositories declares the same repositories as the Bzlmod extension and registers their toolchains, which the Go companion module does by itself under Bzlmod. Checked by hand in a WORKSPACE consumer outside the repository: with the rules_go fork that the Orchestrion loads still need, an instrumented dd_topt_go_test passes, its standard library comes from the instrumented SDK, and only the linux_amd64 SDK is fetched; base_sdk_roots pointing at the registered @go_sdk works too.
INSTRUMENTATION=ddtest runs the positive WORKSPACE and Bzlmod fixtures with the test instrumented by ddtest instrument instead of Orchestrion: - Bzlmod declares topt_go_ddtest.sdk(go_version = ...), and WORKSPACE calls topt_go_ddtest_workspace_repositories with the registered @go_sdk as the base SDK of the host's exec platform. - The test checks the bazel.go.instrumentation.* and bazel.go.sdk.* metadata and that no Orchestrion metadata is left. - The hermetic lane checks with assert_ddtest_aquery.py, in the default, opt and no-strip modes, that the test's compiles, link and standard library use the instrumented SDK, that the compile gets the registration file and the link Mini's testopt library, and that only the TestRunner has no-remote-exec. - Payload, BEP freshness and uploader dry-run checks are the same as Orchestrion's. The Orchestrion-only scenarios (pin files, the tool repository's errors, the disabled and Windows smokes) stay Orchestrion's. CI runs the ddtest fixtures in every rules_go compatibility shard, without the host's Go.
…xture ddtest_host_platform did not know Git Bash's MINGW/MSYS uname, so the Python test that writes the ddtest WORKSPACE failed on Windows.
dd_topt_go_test builds every enabled test with the SDK that ddtest instruments: the instrumentation, orchestrion_mode and orchestrion_pin_files parameters are gone, and passing orchestrion_mode or orchestrion_pin_files fails with a migration message instead of reaching go_test. The test-binary linker flags apply by default, and no-remote-exec always stays on the TestRunner only. The target metadata always carries bazel.go.instrumentation.* and bazel.go.sdk.*, never bazel.go.orchestrion.*. dd_topt_go_stdlib_warmup always warms the instrumented standard library. topt_go_orchestrion.bzl and topt_go_orchestrion_repository.bzl are removed; the launcher tests now cover ddtest_go_test, and the tests that only covered Orchestrion's modes, pin files and transition are removed.
The Go companion and the root module use upstream rules_go from the BCR. Removed: third_party/rgo and third_party/rules_go_orchestrion, the fork registry, materializer, diff, profile, smoke and drift tooling, their tests and CI jobs, and the release archive's fork checks. The WORKSPACE helper datadog_go_test_optimization_workspace_repositories now declares only the Go companion, like the Python and Java helpers: the consumer declares upstream rules_go first, and rules_go_repo_name only maps it. Release notes and release.json drop the fork blocks.
The WORKSPACE and Bzlmod fixtures now only cover ddtest instrument, on upstream rules_go (RULES_GO_VERSION: a BCR bazel_dep, or the release zip with its BCR integrity in WORKSPACE). The Bzlmod fixture also checks that a build of the library alone fetches neither the instrumented SDK nor Mini, and contacts no metadata server while Test Optimization is disabled. WINDOWS_SMOKE_ONLY runs the instrumented test on Windows. The Orchestrion-only scenarios (pin files, the tool repository and its errors, the disabled and config-transition smokes, measure mode) and their helpers are removed. CI: workspace-compat runs rules_go 0.60.0, 0.61.1, 0.62.0, 0.63.0, 0.64.2 and 0.65.0 (with Go 1.26) in both modes without the host's Go; windows-go-smoke replaces the Orchestrion Windows bootstrap lane, the no-host-Go Orchestrion lane goes, and the Go examples build on Windows again.
The example modules configure topt_go_ddtest from the Go module's go.mod instead of pointing at the Orchestrion bootstrap, and drop the Orchestrion flag, orchestrion.tool.go, orchestrion.yml and the dd-trace-go and Orchestrion requirements that only served as pins.
dd_topt_go_bootstrap writes a managed MODULE.bazel block that configures topt_go_ddtest.sdk (go_version from --runtime-version, or go_mod from the Go module's go.mod) instead of the rules_go fork override and the orchestrion extension. Rerunning it replaces the old Orchestrion block, and it stops with a message when an unmanaged rules_go override still points at the removed fork. The WORKSPACE snippet declares upstream rules_go first and calls topt_go_ddtest_workspace_repositories; wrappers no longer pass Orchestrion attributes. Removed with Orchestrion: the dd-trace-go pinning and go.mod sync, the Orchestrion files, the go_repository drift check, the fork and archive overrides and their flags. The onboarding pins keep only the rules_test_optimization commit and archive. The mock-server suite's multi-service scenario uses upstream rules_go and checks the hidden raw test, and its bootstrap scenarios cover the new block, the migration from the old one and a guided instrumented test.
The message pointed WORKSPACE users at topt_go_ddtest_repositories, which registers no toolchains; topt_go_ddtest_workspace_repositories does. The macro's comments name Mini instead of the tracer.
README, CONTRIBUTING, AGENTS and the references describe the Bzlmod and WORKSPACE setup with topt_go_ddtest, upstream rules_go, the bootstrap's managed block, the target metadata, the integration fixtures and CI lanes, and the ddtest failure modes in Troubleshooting. The Go onboarding skill is rewritten for ddtest; the large-monorepo rollout guide keeps its generic checklist. Removed: the Orchestrion and rules_go fork documents and the rules-go-orchestrion-upstream-migration skill. CHANGELOG's Unreleased section lists the changes and how Go consumers migrate.
The registration tool treated every build tag but ignore as set, so a test whose sources all carry a negated constraint such as !windows had no package and failed with "no Go sources". A file is now skipped only when its constraint holds on no platform, as with ignore. The registration rule also reads the sources of the libraries the test embeds, as rules_go compiles them into the test. The Go WORKSPACE helper rejects a rules_go_repo_name other than io_bazel_rules_go: the SDK repositories that ddtest instrument writes load rules_go under that name. Troubleshooting covers a select() in deps that also lists Mini's testopt.
Sources without a build constraint now decide the registration package; constrained ones only decide when every source has one, so a file that the configuration excludes, such as a //go:build tools file of another package, no longer conflicts. When deps is a select(), which the macro cannot inspect, Mini joins it through a private alias: the same archive under another label, so a select that already names testopt no longer duplicates it.
The alias that adds Mini to configurable deps did not carry the test's target_compatible_with, so a wildcard cquery over an incompatible test analyzed it and fetched Mini. It now has the test's constraints, and the helper compatibility fixture and its cquery check cover it. The foreign-Mini analysis error also tells WORKSPACE users what to do, and Troubleshooting describes that instrumented tests ignore an sdk_version transition below them.
Inside an instrumented test every Go target resolves to an instrumented SDK, because the test's transition sets sdk_name to ddto and the toolchains matched only that. A go_cross_binary(sdk_version = ...) below the test, such as a FIPS binary in data, then built silently with the default instrumented SDK. The toolchains now match sdk_version as rules_go matches its own: the default SDK serves an unset value and the values of its go_version (major, major.minor, major.minor.patch and prerelease forms). More topt_go_ddtest.sdk tags with sdk_version, or sdk_variants in WORKSPACE, add SDK variants with their own go_version, base_sdk_roots, exec_platforms and experiments. A stand-in toolchain catches only an sdk_version that no SDK serves and fails with what to configure; it never matches a served value, so it cannot make an exec platform without an SDK eligible. A base SDK whose VERSION is not its go_version now fails with a message that says so. Tests: unit tests for the matching, variant validation, generated toolchains and base SDK check; sdk_variants_test runs data binaries built with the default SDK, a version prefix and a boringcrypto variant on another Go version, and an exec tool that imports testing; the Bzlmod integration checks the unconfigured-sdk_version failure and the WORKSPACE integration a variant and the stale base SDK failure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The next major version of the Go support: Go tests instrumented by
ddtest instrument bazelinstead of Orchestrion, on upstream rules_go.@io_bazel_rules_go//go/toolchain:sdk_name; other builds fetch nothing.pre-release-f5df4db(DataDog/ddtest f5df4db), with the checksums of its executables.orchestrion_modeandorchestrion_pin_files(passing them fails with a migration message).topt_go_ddtest.sdk(go_mod = ... | go_version = ...). WORKSPACE: upstream rules_go, thentopt_go_ddtest_workspace_repositories(...).dd_topt_go_bootstrapwrites and migrates both setups.@io_bazel_rules_go//go/toolchain:sdk_versionlike rules_go's own. Moresdktags withsdk_version(orsdk_variantsin WORKSPACE) add SDK variants, such as a FIPS SDK for ago_cross_binaryin a test'sdata; ansdk_versionthat no SDK serves fails with what to configure.One commit per step. The Orchestrion path kept working until the removal commits, so the payloads below were compared on the same targets. The removal commits (from
3d69889) are split by area and are not green one by one; the PR is meant to be squash-merged.io_bazel_rules_godd_topt_go_test, metadata, stdlib warmupsdk_versionand SDK variants, from a review against a large consumerValidation
cd modules/go && ../../bazelw test //... --override_module=datadog-rules-test-optimization=../..: 160 tests pass.run_go_helper_compatibility.shpasses and now also requires the Mini alias of configurable deps to be incompatible when its test is../bazelw test //tools/... //examples/...: 213 tests; 212 passed in the full run andpython_tools_testtimed out in itspwshlogging subtests under that load, then passed alone. CI runs the same suite.modules/go/tests/ddtest:instrumented_testruns instrumented and fails unless Mini set its context's trace (a transition that does not select the SDK makes it fail);foreign_mini_failure_testexpects the analysis error for a foreign copy of Mini; the stdlib warmup builds with the instrumented SDK.run_bzlmod_go_integration.shandrun_workspace_go_integration.sh, locally withFORBID_HOST_GO=1(ddtest needs no host Go), at both ends of CI's matrix: rules_go 0.60.0 with Go 1.25.0 and rules_go 0.65.0 with Go 1.26.3, in both modes: pass. They check thebazel.go.instrumentation.*andbazel.go.sdk.*metadata; payloads, BEP freshness and the uploader dry run with--validate-enrichment; and, in the hermetic lane, withassert_ddtest_aquery.pyin the default, opt and no-strip modes, that the test's compiles, link and standard library use the instrumented SDK, with the registration file and Mini'stestopt. Bzlmod also checks that building the library alone fetches neither the SDK nor Mini, and contacts no metadata server with Test Optimization disabled. WORKSPACE instruments the registered@go_sdkthroughbase_sdk_roots.run_mock_server_tests.sh: passes (about 11 minutes), including the bootstrap scenarios (new managed block, migration from the Orchestrion block) and a guided test built with the instrumented SDK.sdktag onlytopt_go_ddtest_toolchainsis fetched; with one, only the SDK of the exec platform in use.sdk_version and SDK variants
A read-only review against a large consumer found that a
go_cross_binary(sdk_version = "<version>fips")in an instrumented test'sdatabuilt with the default instrumented SDK: the transition setssdk_nametoddtofor everything below the test, and the toolchains matched only that. The Orchestrion fork turned the instrumentation off on tool and data edges, which upstream rules_go cannot do without patches.sdk_versionas rules_go'sdeclare_bazel_toolchainsdoes: the default SDK serves an unset value and the values of itsgo_version; a variant serves its ownsdk_version. Variants have their owngo_version,base_sdk_roots,exec_platformsandexperiments.sdk_versionthat no SDK serves (an alias whoseselect()negates aconfig_setting_groupof every served value) and fails with the requested value and the configured SDKs. It never matches a served value, so it cannot make an exec platform without an SDK eligible.VERSIONdiffers from itsgo_versionfails at fetch time;go_wrap_sdkalso gets the version, for SDKs without aVERSIONfile.Validation, on Linux amd64 (
0d8394f):modules/go //...163 tests, both integration scripts withFORBID_HOST_GO=1at rules_go 0.60.0 and Go 1.25.0, and//tools/... //examples/...(213 tests;python_tools_testtimed out in itspwshsubtests under load and passed alone) all pass. In particular:modules/go/tests/ddtest:sdk_variants_testruns, from itsdata: a plaingo_binary(go1.26.3, the default SDK), ago_cross_binarywithsdk_version = "1.26"(go1.26.3), one withsdk_version = "1.25.9fips", a variant on Go 1.25.9 withboringcrypto(go1.25.9 X:boringcrypto), and the output of an exec tool that importstesting. With the toolchains'sdk_versionsetting removed, it fails: the FIPS binary printsgo1.26.3.1.26rc2), variant validation, the generated toolchains and stand-in, the base SDK check.run_bzlmod_go_integration.sh: a test whose data asks for an unconfiguredsdk_versionfails to build with the stand-in's message.run_workspace_go_integration.sh: ansdk_variantsentry serves ago_cross_binaryin the test's data, and a variant whose base SDK is not itsgo_versionfails with that message.modules/gowith both SDKs onlinux_amd64only and a fakelinux_arm64exec platform registered first: the build skips the fake platform and runs on the host; with the first version of the stand-in (nosdk_versioncondition), analysis picked the fake platform and failed.sdk_versionvalues; reproduced and checked by hand as abovego_versionthat its toolchains advertiseVERSION, and throughgo_wrap_sdkwithout onemajor.minorselectors1.26rc2Auditor (GPT-6.1-Sol, xhigh, 3 rounds)
//go:build !windows) and failed with "no Go sources"ignore); unconstrained sources decide the package. Unit cases andconstrained_testembedhad no registration packageGoInfo.srcs.embed_only_testdepsas aselect()that already names Mini'stestoptgot a duplicated labelselect()through a private alias (same archive, onetestopt.ain the link) that carries the test'starget_compatible_with.select_deps_testand the helper compatibility fixturerules_go_repo_namein WORKSPACE could not work with the generated SDK repositoriesio_bazel_rules_goand says why//go:build toolsfile ofpackage mainin the test'ssrcs)Known limitations
sdk_versionselects). Its hooks do nothing in programs that do not link Mini. Building tools with the ordinary SDK would need transitions that upstream rules_go does not offer.1428be7on Linux, macOS and Windows; it is being repinned to this head.Payloads compared with Orchestrion
The Bzlmod consumer fixture, built once with Orchestrion (
test_optimizationmode, dd-trace-go v2.9.1) and once with ddtest, on the same targets and mock backend:hello_testand acompare_testwith passing, failing and skipped tests, subtests, parallel tests, examples and an external test package. Events are matched by type, module, suite and name; identifiers, timestamps, durations,runtime-idandprocess_idare ignored. The harness is a local copy of the fixture and stays out of the repository.ExampleGreeting(pass) andExampleWrong(fail); Orchestrion reports no examplesauto_test_retryruns ofTestFail,TestSubtestsand each of its subtests, with the sametest.final_statusvalues. Early flake detection retried no test in either run, so it is not comparedTags on every event:
languagelibrary_name(payload metadata)ddtest_dd.library_capabilities.coverage_report_upload1_dd.tags.process,_dd.profiling.enabled,_dd.top_level,_dd.trace_span_attribute_schema,_sampling_priority_v1error.stacktesting/testing.goexternal/...topt_go_ddtest_mini,GOROOT/src/testing/testing.gotest.working_directorydiffers only because each build ran from its own workspace directory, so the execroots differ. Every other tag and value, statuses and source lines included, is the same. The test logs differ only by dd-trace-go'sloading features: ... localhost:8126error line, which ddtest does not print.Release Readiness
LICENSE-3rdparty.csvwhen dependencies or vendored code changed.