Skip to content

ci/spec22-gems: fix the env -i continuation split that dropped TEBAKO_RUNTIME_IMAGE (incident 13 round 5) - #90

Merged
ronaldtse merged 2 commits into
mainfrom
ci/spec22-runprobe-env-split
Aug 18, 2026
Merged

ci/spec22-gems: fix the env -i continuation split that dropped TEBAKO_RUNTIME_IMAGE (incident 13 round 5)#90
ronaldtse merged 2 commits into
mainfrom
ci/spec22-runprobe-env-split

Conversation

@ronaldtse

Copy link
Copy Markdown
Contributor

Incident 13 round 5 — root cause of the run-32188296332 all-legs failure

The instrumented rerun failed all four jailed legs with a NEW signature:

`RubyGems' were not loaded.
`error_highlight' / `did_you_mean' / `syntax_suggest' were not loaded.
A:/probe/probe.rb:66:in '<main>': undefined method 'clear_paths' for module Gem (NoMethodError)

plus a misleading driver debug line mounted what=failed to mount image ... at=A:/
(that line is the SUCCESS path — what is the error-context string; the payload
mount worked, as the [tfs] open: A:/probe/... lines prove).

Root cause

#89 (e318712) placed a 5-line comment block inside the backslash-continued
env -i list in run_probe. A comment line inside a continuation terminates the
command at that point (the \ splices the comment line in; the comment ends at its
own newline), so run_probe executed two commands:

  1. env -i MSYS2_... TEBAKO_RUNTIME_IMAGE=... — no command argument: prints the
    env and exits 0, invisible under set +e.
  2. TEBAKO_JAIL=... TEBAKO_DEBUG_TFS=1 "$RUNTIME_EXE" --tebako-image ... — running
    with the ambient msys environment and without TEBAKO_RUNTIME_IMAGE,
    TEBAKO_HOME, HOME, APPDATA, USERPROFILE, TMP*, the System* baseline,
    and the env scrub.

With TEBAKO_RUNTIME_IMAGE unset the driver early-returns and the env image never
mounts, so rubygems and the default gems cannot load — exactly the prelude warnings
above — and gem_prelude's partial Gem module dies at probe.rb:66. The
TEBAKO_HOME loss also explains the r9 artifact's missing journal.log (the store
journal went to the runner's real ~/.tebako; the upload path was correct).

Verified locally with a same-shape repro: comment-in-list splits the command
(ambient env leaks, listed vars lost); comment-outside delivers every assignment
under the scrub. bash -n clean. The one other comment-after-continuation site in
ci/ (run-msys.sh:94) is inside a comment block — harmless.

tebako#417 exonerated

Its traces are pure diagnostics; the r9 logs show the closure-walk verdicts working
(styles-tree materialize cache hits). The traces + this flag's passthrough are
exactly what made the mis-parse diagnosable in one read.

The tripwire (second commit)

probe.rb now prints the three handoff var presences (TEBAKO_RUNTIME_IMAGE,
TEBAKO_HOME, HOME) immediately after the PROBE gem-loaded canary — pure ENV
reads, no rubygems dependency, never gates. A lost handoff names itself in the
first lines of every leg's proof log instead of a full CI cycle later.

Expected effect on the factory dogfood

Restores the r8 environment AND keeps the TEBAKO_DEBUG_TFS traces — the next
factory run should show the r8 leg outcomes plus the closure-walk [tfs] verdicts
for the original sassc 126, which round 4 could not see.

… env -i continuation (incident 13 round 5)

The round-4 diagnostics insert (e318712) placed a 5-line comment block
INSIDE the backslash-continued env -i list. A comment line in a
continuation terminates the command there: the line above splices the
comment in, the comment ends at its own newline, and the remaining
lines parse as a NEW command. run_probe therefore executed TWO commands:

  1. env -i MSYS2_... TEBAKO_RUNTIME_IMAGE=...   (no command: prints
     the env and exits 0 — invisible under set +e)
  2. TEBAKO_JAIL=... TEBAKO_DEBUG_TFS=1 "$RUNTIME_EXE" --tebako-image ...
     — with the AMBIENT msys environment, without the env -i scrub, and
     without TEBAKO_RUNTIME_IMAGE / TEBAKO_HOME / HOME / APPDATA /
     USERPROFILE / TMP* / the System* baseline.

The driver early-returns when TEBAKO_RUNTIME_IMAGE is unset, so the env
image never mounted: rubygems and the default gems failed to load (the
prelude warnings in every r9 proof log), gem_prelude's partial Gem
module survived, and probe.rb died at line 66 (Gem.clear_paths,
NoMethodError) on all four legs of run 32188296332. The TEBAKO_HOME
loss also redirected the store journal to the runner's real ~/.tebako —
the r9 artifact's missing journal.log was a symptom, not an upload bug.

tebako#417 is exonerated: its traces are pure diagnostics, and the r9
logs show the materialize verdicts working (styles-tree cache hits).
The traces plus this flag's passthrough are exactly what made the
mis-parse diagnosable.

Verified: bash -n clean; a same-shape repro shows comment-in-list
splits the command (ambient env leaks, listed vars lost) while the
comment-outside shape delivers every assignment under the scrub.
…ames itself

Incident 13 round 5's failure was invisible from inside the payload:
the runtime booted without TEBAKO_RUNTIME_IMAGE/TEBAKO_HOME/HOME and
the only payload-side symptom was the rubygems prelude warning plus the
line-66 NoMethodError, a full CI cycle removed from the cause. Print
the three var presences right after the gem-loaded canary (pure ENV
reads, no rubygems dependency), so a broken handoff shows in the first
lines of every leg's proof log. Never gates.
@ronaldtse
ronaldtse marked this pull request as ready for review August 18, 2026 23:44
@ronaldtse
ronaldtse merged commit f043e0d into main Aug 18, 2026
42 checks passed
@ronaldtse
ronaldtse deleted the ci/spec22-runprobe-env-split branch August 18, 2026 23:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants