Skip to content

fix(runtime): stop journaling a vendor's own terminal-reply storm as out-of-band input - #153

Merged
nikolasd merged 7 commits into
mainfrom
echo-storm
Sep 9, 2026
Merged

nikolasd merged 7 commits into
mainfrom
echo-storm

Conversation

@nikolasd

@nikolasd nikolasd commented Sep 9, 2026

Copy link
Copy Markdown
Owner

A vendor TUI's own redraw-driven escape queries were being journaled as if a human had typed them. crewd attach's viewer socket carries genuine keystrokes and a real terminal's replies to the vendor's own queries on the same duplex stream, and serve_viewer treated every byte read on it as out-of-band input regardless of content: a run left attached and idle in a real terminal (measured against a live tmux pane, which is confirmed -- via display/tmux.rs's own placement code -- to be the real terminal-emulator-in-the-loop for production attach panes) produced 99 outOfBandInput journal events and 99 needsReconciliation flag-sets in 20 seconds of pure idle, out of 211 total events for the run. The reported incident (a run left attached for eight minutes) journaled over 7,300 rows for a single run.

The byte source, confirmed before any fix was written

Reproduced headlessly: a fake vendor process emitting the CPR (ESC[6n) and primary-DA (ESC[c) queries a redrawing TUI issues, attached via crewd attach hosted in a real tmux pane. Every byte arriving on the viewer socket matched exactly the pattern ESC[<row>;<col>R (CPR reply) immediately followed by ESC[?1;2;4c (primary DA reply) -- tmux answering the vendor's queries on its behalf, the only path that could ever answer them, since crew's own attach relay never interprets terminal escape codes. Zero printable keystroke bytes anywhere in the capture.

The fix

display::terminal_reply recognizes the fixed shapes a real terminal emits automatically and a human keyboard never does: CPR replies, primary/secondary DA replies, the DSR "terminal OK" reply, focus in/out reports, and bracketed-paste start/end markers (the markers only -- text pasted between them is untouched and still counts as real input). serve_viewer strips these before deciding whether to call on_user_input at all; a read that is nothing but replies never reaches it, so it neither journals an event nor sets needsReconciliation. The vendor's own PTY still receives every byte unfiltered, exactly as before -- this is the only path that answers the vendor's own queries, so forwarding must not change.

A socket read can split a reply sequence across two reads. This was measured live, not theorized: the committed reproduction test (below) failed intermittently (roughly 30-50% of runs) after the coalescing work landed, with a spurious single-input row appearing. Root cause: a lone trailing c in one read, and a truncated ESC[?1;2;4 missing its closing c arriving in the next. ReplyFilter carries up to 32 unresolved trailing bytes across reads within one connection, so a split sequence is recognized once the next read completes it, rather than leaking either half. Regression tests split the captured reply at every possible byte offset, and chunk a 146-reply burst at a 7-byte boundary that doesn't divide the reply's 17-byte length.

Holding a byte back is driven by volume (more bytes arriving), not time. On its own this would hold a lone Escape keypress forever if nothing else ever arrived -- and Escape, alone, interrupts a claude TUI's current turn, about as consequential an out-of-band action as exists. serve_viewer's read loop also races an idle-window sleep (shared with the coalescer's own IDLE_WINDOW rather than a second timer, re-armed every iteration, enabled only while something is held) that releases whatever's pending as content once a full idle window passes with no further read -- through the same on_user_input path an ordinary keystroke takes, so a released byte coalesces like any other input rather than journaling a row of its own.

Coalescing: OutOfBandInput now carries input_count: u64 (#[serde(default = "one_input")], defaulting to 1 -- an outOfBandInput event means a human typed, so a pre-coalescing row cannot truthfully claim zero) and span_ms: u64 (#[serde(default)], defaulting to 0 -- a pre-coalescing row is a single input spanning no time), both #[ts(type = "number")], both plain u64 and never String so neither needs a redaction-guard allowlist entry -- content-free by type, not just by review. A background coalescer batches rapid on_user_input firings and flushes one row after a second of quiet, rather than one row per read. The variant's doc comment states it is journaled at the end of the window it describes, so the window's start is reconstructible from the envelope timestamp minus span_ms without a second, redundant timestamp field.

Also fixed: a transcript format that stamps every entry with its session id (Claude's does) was re-deriving and re-journaling an identical AdapterVendorSessionEvent for every tailed entry -- 41 identical rows for one session id, measured on a real transcript. The pump loop now tracks the last vendor session id it actually emitted and skips a redundant re-emit, but only when doing so has no effect on cursor durability: the common shape (SessionMeta paired with another event on the same transcript line) already gives SessionMeta no cursor, since the sibling event carries it, so skipping it is free; the rarer "hidden SessionMeta-only line" shape still emits even when redundant, since dropping that call would drop the only commit advancing runs.transcript_cursor past it.

Verification

  • The committed reproduction test drives the real production path end to end: a fake vendor emitting the exact captured query pair every 50ms under a real tmux pane hosting crewd attach. A positive control (tmux capture-pane -S -, counting DA-reply occurrences with a floor of 20) runs and must pass before the real assertion, so a scaffold producing nothing cannot pass the zero-journaled-rows check for the wrong reason -- verified the floor itself fires by temporarily breaking it and confirming the test failed with the right message. Skips visibly when tmux isn't on PATH, rather than failing.
  • cargo fmt --all --check -- clean.
  • cargo clippy --all-targets --all-features -D warnings -- clean.
  • CREW_DISABLE_VENDOR_CLI=1 cargo test --workspace -- all green, including 17/17 terminal_reply tests, 3/3 oob_coalescer tests, and the storm reproduction test run four times total across this review with zero flakes (post-fix).
  • bun run check -- exit 0 end to end (schema-drift, format, typecheck, build, bun test, full Rust workspace).
  • bun run generate re-run for the OutOfBandInput schema/bindings change.

nikolasd and others added 7 commits September 9, 2026 22:30
…ranscript entry

A transcript format that stamps every entry with its session id
(Claude's does) re-derived a TuiEvent::SessionMeta -> VendorSessionEstablished
mapping for every single tailed entry, journaling an identical row each
time -- 41 identical rows for one session id, measured on a real
transcript during the echo-storm investigation.

The pump loop now tracks the last vendor session id it actually
emitted (seeded from the initial best-effort guess derived from the
transcript path) and skips a SessionMeta whose id hasn't changed --
but only when doing so has no effect on cursor durability. A
SessionMeta paired with another event on the same transcript line (the
common shape, e.g. SessionMeta + AssistantText) already carries no
cursor from cursor_placements, since the other event is the carrier,
so skipping it is free. The rarer 'hidden SessionMeta-only line' shape
(nothing else on the line to carry the cursor) still emits even when
redundant, since dropping that call would drop the only commit that
advances runs.transcript_cursor past it -- a resume must never skip
that.
…out-of-band input

A vendor TUI's redraw-driven cursor-position (CPR) and device-attributes
(DA) escape queries, relayed through crewd attach's mirrored output to
whatever real terminal is watching (a human's, or -- confirmed in
production -- tmux hosting the attach pane), come back as replies on
the same socket ordinary keystrokes arrive on. serve_viewer treated
every byte read on that socket as out-of-band input regardless of
content, journaling an outOfBandInput event and setting
needsReconciliation on every single reply. Measured live: idling 20
seconds with nobody typing produced 99 such events against 211 total
-- an append-only journal growing 60x in eight minutes on the run that
surfaced this, and a reconciliation flag firing on noise.

Adds display::terminal_reply, a small filter recognizing the fixed
shapes a real terminal emits automatically and a human keyboard never
does: CPR replies, primary/secondary DA replies, the DSR terminal-OK
reply, focus in/out reports, and bracketed-paste start/end markers
(the markers only -- pasted text between them is untouched and still
counts as real input). serve_viewer now strips these before deciding
whether to call on_user_input at all; a read that is nothing but
replies never journals anything or touches needsReconciliation. The
vendor's own PTY still receives every byte unfiltered, exactly as
before -- this is the only path that ever answers the vendor's own
queries, so forwarding must not change.

Adds a committed integration test reproducing the storm end to end
through the real production path: a fake vendor emitting the exact
captured query pair every 50ms under a real tmux pane hosting crewd
attach (tmux is confirmed, via display/tmux.rs's own placement code,
to be the real terminal-emulator-in-the-loop for production attach
panes -- a bare pty pair with nobody driving it does not auto-answer
these queries, so this cannot be reproduced any other way). Skips
visibly when tmux isn't on PATH rather than failing.

Also stops re-journaling AdapterVendorSessionEvent (VendorSessionEstablished)
on every transcript entry for a format that stamps each one with its
session id (Claude's does) -- 41 identical rows for one id, measured on
a real transcript during the same investigation. Only a change reaches
the sink now, gated so it never suppresses the rare case where a
redundant SessionMeta is the sole carrier of its line's cursor (see the
inline doc comment on the pump loop for why that carve-out is
load-bearing for crash-resume correctness).
…idle window

OutOfBandInput's payload now carries input_count and span_ms: how many
out-of-band inputs one row coalesces, and the span between the first
and the last. A background coalescer (adapter::tui::oob_coalescer)
batches rapid on_user_input firings and flushes one row after 1 second
of quiet, rather than journaling one row per read as before -- this
covers genuine rapid human typing directly into an attached pane, the
case the terminal-reply filter's own fix does not touch (that filter
already keeps a vendor's own reply storm from reaching on_user_input
at all).

input_count defaults to 1, not 0, on rows written before coalescing
existed: an OutOfBandInput event means a human typed, so a row cannot
truthfully claim zero. span_ms defaults to 0: a pre-coalescing row is a
single input, spanning no time. Both are plain u64 fields, never
String, so neither needs a redaction-boundary allowlist entry -- content-free by
type, not just by review. RuntimeEvent::OutOfBandInput's own doc
comment states it is journaled at the end of the window it describes,
so the window's start is reconstructible from the envelope timestamp
minus span_ms without a second, redundant timestamp field.

Fixes a real bug the coalescing work's own reproduction test caught:
strip_terminal_replies operated on one socket read at a time, and a
read can split a reply sequence in two (measured live -- a lone
trailing 'c', or a truncated escape sequence missing its closing byte
in the next read), leaking a spurious byte or two as residue and
occasionally journaling one stray outOfBandInput row per test run.
display::terminal_reply::ReplyFilter now carries up to 32 unresolved
trailing bytes across reads within one connection, so a split sequence
is recognized once the next read completes it rather than leaking
either half. bun run generate re-run for the schema/bindings change.
…ot only by volume

ReplyFilter held a trailing, possibly-incomplete escape sequence back
across reads, but only released it once enough further data arrived
to push it out of the lookback window -- a purely volume-driven
mechanism. A lone Escape keypress with nothing typed after it (a real,
consequential out-of-band action: it interrupts a claude TUI's current
turn) has no such volume to age it out by, so it would have been held
back forever, never journaled, never setting needsReconciliation.

serve_viewer's read loop now also races an idle-window sleep,
re-armed every iteration and only enabled while the filter is
holding something back; when nothing else arrives before it fires, it
takes whatever is pending and releases it as content. Shares
adapter::tui::oob_coalescer's own IDLE_WINDOW constant (now
pub(crate)) rather than inventing a second timer value to keep in
sync with it.

Renamed the one existing test that exercised release-by-volume, since
'ages out unresolved' reads as a timer and would have hidden this gap
from a reader who trusted the name. Added a test for the case none of
the existing ones covered: a lone ESC released with no further bytes
ever supplied, via the caller-facing has_pending/take_pending pair
directly (the actual timer-driven wakeup lives in serve_viewer's async
loop, not in ReplyFilter itself, so this is what a unit test can
exercise).
@nikolasd
nikolasd merged commit 531fb5e into main Sep 9, 2026
19 checks passed
@nikolasd
nikolasd deleted the echo-storm branch September 9, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant