Repository navigation
fix(runtime): stop journaling a vendor's own terminal-reply storm as out-of-band input - #153
Merged
Merged
Conversation
…ranscript entry A transcript format that stamps every entry with its session id (Claude's does) re-derived a TuiEvent::SessionMeta -> VendorSessionEstablished mapping for every single tailed entry, journaling an identical row each time -- 41 identical rows for one session id, measured on a real transcript during the echo-storm investigation. The pump loop now tracks the last vendor session id it actually emitted (seeded from the initial best-effort guess derived from the transcript path) and skips a SessionMeta whose id hasn't changed -- but only when doing so has no effect on cursor durability. A SessionMeta paired with another event on the same transcript line (the common shape, e.g. SessionMeta + AssistantText) already carries no cursor from cursor_placements, since the other event is the carrier, so skipping it is free. The rarer 'hidden SessionMeta-only line' shape (nothing else on the line to carry the cursor) still emits even when redundant, since dropping that call would drop the only commit that advances runs.transcript_cursor past it -- a resume must never skip that.
…out-of-band input A vendor TUI's redraw-driven cursor-position (CPR) and device-attributes (DA) escape queries, relayed through crewd attach's mirrored output to whatever real terminal is watching (a human's, or -- confirmed in production -- tmux hosting the attach pane), come back as replies on the same socket ordinary keystrokes arrive on. serve_viewer treated every byte read on that socket as out-of-band input regardless of content, journaling an outOfBandInput event and setting needsReconciliation on every single reply. Measured live: idling 20 seconds with nobody typing produced 99 such events against 211 total -- an append-only journal growing 60x in eight minutes on the run that surfaced this, and a reconciliation flag firing on noise. Adds display::terminal_reply, a small filter recognizing the fixed shapes a real terminal emits automatically and a human keyboard never does: CPR replies, primary/secondary DA replies, the DSR terminal-OK reply, focus in/out reports, and bracketed-paste start/end markers (the markers only -- pasted text between them is untouched and still counts as real input). serve_viewer now strips these before deciding whether to call on_user_input at all; a read that is nothing but replies never journals anything or touches needsReconciliation. The vendor's own PTY still receives every byte unfiltered, exactly as before -- this is the only path that ever answers the vendor's own queries, so forwarding must not change. Adds a committed integration test reproducing the storm end to end through the real production path: a fake vendor emitting the exact captured query pair every 50ms under a real tmux pane hosting crewd attach (tmux is confirmed, via display/tmux.rs's own placement code, to be the real terminal-emulator-in-the-loop for production attach panes -- a bare pty pair with nobody driving it does not auto-answer these queries, so this cannot be reproduced any other way). Skips visibly when tmux isn't on PATH rather than failing. Also stops re-journaling AdapterVendorSessionEvent (VendorSessionEstablished) on every transcript entry for a format that stamps each one with its session id (Claude's does) -- 41 identical rows for one id, measured on a real transcript during the same investigation. Only a change reaches the sink now, gated so it never suppresses the rare case where a redundant SessionMeta is the sole carrier of its line's cursor (see the inline doc comment on the pump loop for why that carve-out is load-bearing for crash-resume correctness).
…idle window OutOfBandInput's payload now carries input_count and span_ms: how many out-of-band inputs one row coalesces, and the span between the first and the last. A background coalescer (adapter::tui::oob_coalescer) batches rapid on_user_input firings and flushes one row after 1 second of quiet, rather than journaling one row per read as before -- this covers genuine rapid human typing directly into an attached pane, the case the terminal-reply filter's own fix does not touch (that filter already keeps a vendor's own reply storm from reaching on_user_input at all). input_count defaults to 1, not 0, on rows written before coalescing existed: an OutOfBandInput event means a human typed, so a row cannot truthfully claim zero. span_ms defaults to 0: a pre-coalescing row is a single input, spanning no time. Both are plain u64 fields, never String, so neither needs a redaction-boundary allowlist entry -- content-free by type, not just by review. RuntimeEvent::OutOfBandInput's own doc comment states it is journaled at the end of the window it describes, so the window's start is reconstructible from the envelope timestamp minus span_ms without a second, redundant timestamp field. Fixes a real bug the coalescing work's own reproduction test caught: strip_terminal_replies operated on one socket read at a time, and a read can split a reply sequence in two (measured live -- a lone trailing 'c', or a truncated escape sequence missing its closing byte in the next read), leaking a spurious byte or two as residue and occasionally journaling one stray outOfBandInput row per test run. display::terminal_reply::ReplyFilter now carries up to 32 unresolved trailing bytes across reads within one connection, so a split sequence is recognized once the next read completes it rather than leaking either half. bun run generate re-run for the schema/bindings change.
…ot only by volume ReplyFilter held a trailing, possibly-incomplete escape sequence back across reads, but only released it once enough further data arrived to push it out of the lookback window -- a purely volume-driven mechanism. A lone Escape keypress with nothing typed after it (a real, consequential out-of-band action: it interrupts a claude TUI's current turn) has no such volume to age it out by, so it would have been held back forever, never journaled, never setting needsReconciliation. serve_viewer's read loop now also races an idle-window sleep, re-armed every iteration and only enabled while the filter is holding something back; when nothing else arrives before it fires, it takes whatever is pending and releases it as content. Shares adapter::tui::oob_coalescer's own IDLE_WINDOW constant (now pub(crate)) rather than inventing a second timer value to keep in sync with it. Renamed the one existing test that exercised release-by-volume, since 'ages out unresolved' reads as a timer and would have hidden this gap from a reader who trusted the name. Added a test for the case none of the existing ones covered: a lone ESC released with no further bytes ever supplied, via the caller-facing has_pending/take_pending pair directly (the actual timer-driven wakeup lives in serve_viewer's async loop, not in ReplyFilter itself, so this is what a unit test can exercise).
…e) before the echo-storm gate
…trailers) before the echo-storm gate
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A vendor TUI's own redraw-driven escape queries were being journaled as if a human had typed them.
crewd attach's viewer socket carries genuine keystrokes and a real terminal's replies to the vendor's own queries on the same duplex stream, andserve_viewertreated every byte read on it as out-of-band input regardless of content: a run left attached and idle in a real terminal (measured against a live tmux pane, which is confirmed -- viadisplay/tmux.rs's own placement code -- to be the real terminal-emulator-in-the-loop for production attach panes) produced 99outOfBandInputjournal events and 99needsReconciliationflag-sets in 20 seconds of pure idle, out of 211 total events for the run. The reported incident (a run left attached for eight minutes) journaled over 7,300 rows for a single run.The byte source, confirmed before any fix was written
Reproduced headlessly: a fake vendor process emitting the CPR (
ESC[6n) and primary-DA (ESC[c) queries a redrawing TUI issues, attached viacrewd attachhosted in a real tmux pane. Every byte arriving on the viewer socket matched exactly the patternESC[<row>;<col>R(CPR reply) immediately followed byESC[?1;2;4c(primary DA reply) -- tmux answering the vendor's queries on its behalf, the only path that could ever answer them, since crew's own attach relay never interprets terminal escape codes. Zero printable keystroke bytes anywhere in the capture.The fix
display::terminal_replyrecognizes the fixed shapes a real terminal emits automatically and a human keyboard never does: CPR replies, primary/secondary DA replies, the DSR "terminal OK" reply, focus in/out reports, and bracketed-paste start/end markers (the markers only -- text pasted between them is untouched and still counts as real input).serve_viewerstrips these before deciding whether to callon_user_inputat all; a read that is nothing but replies never reaches it, so it neither journals an event nor setsneedsReconciliation. The vendor's own PTY still receives every byte unfiltered, exactly as before -- this is the only path that answers the vendor's own queries, so forwarding must not change.A socket read can split a reply sequence across two reads. This was measured live, not theorized: the committed reproduction test (below) failed intermittently (roughly 30-50% of runs) after the coalescing work landed, with a spurious single-input row appearing. Root cause: a lone trailing
cin one read, and a truncatedESC[?1;2;4missing its closingcarriving in the next.ReplyFiltercarries up to 32 unresolved trailing bytes across reads within one connection, so a split sequence is recognized once the next read completes it, rather than leaking either half. Regression tests split the captured reply at every possible byte offset, and chunk a 146-reply burst at a 7-byte boundary that doesn't divide the reply's 17-byte length.Holding a byte back is driven by volume (more bytes arriving), not time. On its own this would hold a lone Escape keypress forever if nothing else ever arrived -- and Escape, alone, interrupts a claude TUI's current turn, about as consequential an out-of-band action as exists.
serve_viewer's read loop also races an idle-window sleep (shared with the coalescer's ownIDLE_WINDOWrather than a second timer, re-armed every iteration, enabled only while something is held) that releases whatever's pending as content once a full idle window passes with no further read -- through the sameon_user_inputpath an ordinary keystroke takes, so a released byte coalesces like any other input rather than journaling a row of its own.Coalescing:
OutOfBandInputnow carriesinput_count: u64(#[serde(default = "one_input")], defaulting to 1 -- anoutOfBandInputevent means a human typed, so a pre-coalescing row cannot truthfully claim zero) andspan_ms: u64(#[serde(default)], defaulting to 0 -- a pre-coalescing row is a single input spanning no time), both#[ts(type = "number")], both plainu64and neverStringso neither needs a redaction-guard allowlist entry -- content-free by type, not just by review. A background coalescer batches rapidon_user_inputfirings and flushes one row after a second of quiet, rather than one row per read. The variant's doc comment states it is journaled at the end of the window it describes, so the window's start is reconstructible from the envelope timestamp minusspan_mswithout a second, redundant timestamp field.Also fixed: a transcript format that stamps every entry with its session id (Claude's does) was re-deriving and re-journaling an identical
AdapterVendorSessionEventfor every tailed entry -- 41 identical rows for one session id, measured on a real transcript. The pump loop now tracks the last vendor session id it actually emitted and skips a redundant re-emit, but only when doing so has no effect on cursor durability: the common shape (SessionMetapaired with another event on the same transcript line) already givesSessionMetano cursor, since the sibling event carries it, so skipping it is free; the rarer "hiddenSessionMeta-only line" shape still emits even when redundant, since dropping that call would drop the only commit advancingruns.transcript_cursorpast it.Verification
crewd attach. A positive control (tmux capture-pane -S -, counting DA-reply occurrences with a floor of 20) runs and must pass before the real assertion, so a scaffold producing nothing cannot pass the zero-journaled-rows check for the wrong reason -- verified the floor itself fires by temporarily breaking it and confirming the test failed with the right message. Skips visibly whentmuxisn't onPATH, rather than failing.cargo fmt --all --check-- clean.cargo clippy --all-targets --all-features -D warnings-- clean.CREW_DISABLE_VENDOR_CLI=1 cargo test --workspace-- all green, including 17/17terminal_replytests, 3/3oob_coalescertests, and the storm reproduction test run four times total across this review with zero flakes (post-fix).bun run check-- exit 0 end to end (schema-drift, format, typecheck, build, bun test, full Rust workspace).bun run generatere-run for theOutOfBandInputschema/bindings change.