Repository navigation
Prevent truncated command-runner profiles from failing metrics CI #275
Description
Description
The post-merge main metrics job at 7eb98e1 passed its instrumented tests, then strict LLVM profile merging failed. The retained command_runner raw profile is 4,096 bytes, SHA-256 70b183db8845d008dc92b287c1068006ce3e97ef5eba6087c334b6cb604a9c65. Its valid version-11 header declares 21,800 bytes; it ends inside metadata before any counters. Build ID and retained metadata match 14 complete profiles whose symbols establish the command-runner fixture module.
Termination during exit-profile writing is a source-backed hypothesis, not an established historical cause. One unchanged instrumented scoped-parent run on air passed and produced valid profiles. The missing evidence is the owned child's mode, termination status, and profile-write ordering on Linux. The retained artifact includes the raw bytes and explicitly incomplete producer mapping. This follows PR #274.
Why
A truncated synthetic-process profile makes main CI fail without producing a coverage or complexity comparison. Retrying or discarding it would conceal a missing measurement.
Goal
Establish and repair the fixture/profile lifecycle failure while preserving production-runner coverage, output caps, deadlines, and owned-process cleanup.
Acceptance Criteria
- A bounded, credential-free Linux fixture observation or deterministic write-boundary regression distinguishes interrupted profile writing from a write error. Ownership and attribution limits are explicit.
- The supported failure mechanism has a targeted fix and regression. All resulting coverage inputs remain subject to strict parsing; no corrupt profile is ignored, repaired, filtered, or retried.
- Existing finite-output, overflow, timeout, process-tree cleanup, scoped environment/cwd and exit-status assertions remain effective.
- Coverage scope, exclusions, thresholds, comparison baseline and production behavior remain unchanged unless a separately justified defect requires a reviewed change.
Definition of Done
- A follow-up PR starts from current main, preserves the failed artifacts, and documents established cause versus historical uncertainty.
- Current-head Codex review is clean, review threads resolved, and normal Rust, security, mutation and strict metrics checks pass.
- No Home/Laptop, live-provider/account, UI, deployment or task operations are used for qualification.
Validation
- Run the credential-free fixture with the pinned instrumenter on an inert Linux CI host. Retain exact source/tool/binary identities, bounded owned-process/write evidence, raw profiles and strict
llvm-profdata show/mergeresults. - Demonstrate the regression fails before the fix and passes afterward without weakening original fixture assertions or measurement gates.
- Run
cargo fmt --all --check,cargo clippy --workspace --all-targets --locked -- -D warnings,cargo test --workspace --locked, the declared script tests, and normal hosted CI. No UI change is planned.
Activity
HemSoft commentedon Oct 27, 2026
Post-merge follow-through: the main run 40436109242 failed while collecting the historical base
7eb98e1c801c20472cd4944dea8cc7697918bedf, before the fixed candidate was invoked. The retained manifest sayscollection=base,phase=coverage,capture_status=complete, with no candidate directory. The rejected raw belongs to the old fixture module. The workflow selects the push event before-SHA as its base.The merged
1690d015ef554f88156c0b186e53f3e13e3c08f2tree and reviewedab7e8caa475713cd9ef020cba96f2bdda7ec41d1tree are identical (7dd71168203bf39c7678e5019f60ea77f96cc386). The PR-head hosted Linux write-boundary controls, strict production merge, and unchanged comparison passed as documented above. This post-merge result supplies no new candidate measurement and does not demonstrate recurrence in the fixed fixture.Evidence archive SHA-256:
a0e54ea46ca30be1f17cc7c255915ae9cc0a5552e24182d4c8c7b75d347ea6cd. No profile was discarded or repaired; no baseline was rewritten and no run retried. Terminal result at 22:46 EDT: all six other jobs passed, including Windows/macOS/Ubuntu Rust tests, build, security, and mutations. Main still points to1690d015ef554f88156c0b186e53f3e13e3c08f2. The run remains failed because the historical base collection failed; there is no candidate metrics result from this run.HemSoft commentedon Nov 5, 2026
Reopening for a recurrence on main
d213f24be8c939bdbe345fce371cd7efd0e88ef3, after #278: PR #303 base metrics collection passed its tests, then strict LLVM merging rejected a truncated 4,096-byte profile. Original failure evidence is preserved unchanged (SHA-256c3b211f95d6f8e37e98162659731b62b015fc54a040b4c26cffc1327eaf839f3). No candidate comparison was produced.The newly established exit path is in the synthetic overflow fixture:
collect()returns an error and drops the pipe readers before the runner kills the group. The fixture's other writer can receiveBrokenPipe;emit()unwraps that error, panics, and exits through LLVM's profile writer. #278 waits afteremit(), so it does not cover a panic insideemit().A standalone inert Linux replica built with CI's rustc
1a2c4797e/ LLVM 23.1.4 establishes the mechanism: closing pipes without killing produced exit 101 and complete profiles in 100/100 old-fixture runs; tolerating write errors kept 30/30 children idle without a child profile. Delaying the kill reproduced truncated profiles rejected by the exact pinnedllvm-profdatawith the same corrupt-header error. The parent independently repeated five old/modified runs each with the same red/green result. These are replica observations; the exact hosted close-to-kill timing was not observed. CI log ordering and complete-profile counters attribute the remaining bad PID to the overflow slot by elimination, rather than by identity encoded in the short raw file.Repair scope: keep finite
emit()strict; make only held/overflow output tolerateBrokenPipeand remain idle, with a deterministic closed-pipes regression and instrumented write-boundary control. Preserve existing runner assertions, strict profile parsing, scope, thresholds, baseline, production runner behavior, caps, deadlines and cleanup.The repair will be a separate PR from current main. Its old-base collection remains exposed to the old fixture; candidate code cannot repair that base. After it lands, update #303 onto fixed main and verify the job's actual
METRICS_BASE; then advance stacked #304 and qualify all required CI and triggered Package jobs. No blind rerun, profile suppression or baseline substitution is being used.test: keep overflow fixtures idle after broken pipes (#275) (#305)