Skip to content

needs-design: act-gated LLM pane-state classifier for the in-session states no structured channel covers (#727/#739/#680 residual) #746

Description

@pbean

Problem

#727, #739, and #680 are one family: "waiting on a human", "working quietly", and "wedged" are semantically distinct states that produce identical observables at the orchestrator's keyhole (pane-log (mtime_ns, size) via _log_activity_key, pane silence, window liveness). Every deterministic rule at that spot is lossy at the margins, not merely unfinished — #680's own counter-example is a blocked 59-minute docker run being indistinguishable from a deliberate sleep-poll, and #617/#610 established that pane text is model-authored, so vocabulary-based detection collides with healthy story output by construction.

Structured hook events (#745) are the first line and settle the family deterministically wherever the CLI emits them. This issue covers the residual margin: profiles whose hook maps end at Stop, and states no CLI exposes as an event.

Proposal sketch — LLM as untrusted sensor, never authority

This deliberately touches the "no LLM calls in the control loop" invariant, so the design must sharpen the rule rather than erode it. Proposed restatement: the control loop gains no LLM authority. Concretely:

  • Consult points: only at deterministic decision points that already exist — stall-grace expiry pre-nudge in BaseGenericAdapter.wait_for_completion, and pre-crashed verdict. Never on the hot path, never on a timer of its own.
  • Input: the last screenful of the pane (bounded capture), nothing else. No conversation history, no repo access.
  • Output: a closed enum — waiting_on_human | working | wedged | unknown — schema-validated; any free text goes only to the journal.
  • Act-gate (the actual security boundary): a verdict may only narrow toward safer actions: withhold STALL_NUDGE_TEXT, extend the grace, or pause + escalate. It may never authorize keystrokes, a retry, a kill, teardown, or completion. Session completion remains hook-Stop / window-death only (PR fix(adapters): complete dev sessions only on Stop or window death #53 invariant untouched). A waiting_on_human verdict pausing instead of verdicting crashed is what spares StoryTask.attempt (A dev session waiting on an interactive permission prompt is recorded as crashed, and retry burns the last attempt #727's burn sites: Engine._dev_phase, Engine._fix_phase).
  • Failure direction: a false waiting_on_human on a genuinely dead session parks the run instead of retrying — degraded liveness, not damage — and is bounded: N consecutive parked verdicts force the escalation path.
  • Unavailability: LLM unreachable / timeout / quota ⇒ deterministic fallback = today's exact behavior. The classifier must never be load-bearing for control-plane liveness; the orchestrator frequently runs precisely when the LLM side is broken (Dev session lost to an API/transport failure is charged as a story timeout, exhausting the attempt budget #194, Seed a Claude Code usage-limit/quota env-fault pattern once a real line is captured #610).
  • Threat model: pane bytes are session-authored and therefore attacker-influenced; the classifier is prompt-injectable by construction. The closed enum + act-gate is the boundary, not the prompt. A hostile session can at worst talk itself into a pause — never into an action.
  • Cost: bounded consults per session (policy knob), off by default.

Open questions (why needs-design)

  • What runs the consult: a headless one-shot invocation of the run's own adapter binary? A fixed profile? How do the zero-token invariants for live/E2E tests hold (tests must stub it — same posture as the mock adapter)?
  • Policy surface: field names, defaults in data/settings/core.toml (sync test), per-run cap.
  • Journal/TUI shape for verdicts, so a withheld nudge is visible rather than a new silent degrade.
  • How much of [BUG] Stall detection cannot see a session idling inside a tool call: pane-log re-arm keeps the grace alive through sleep #680 remains once the structured idle_prompt and zero-token legs land — this classifier should only be built for whatever margin measurably survives those.

Non-goals

No LLM anywhere near merges, teardown, sweep decisions, verify.py, completion detection, or any repair write. Those stay deterministic unconditionally.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Robustness, enhancement, tests, or docs worth schedulingarea:adaptersCoding-CLI adapters and profilesarea:engineOrchestrator engine and run lifecycleenhancementNew feature or requestneeds-designAwaiting a maintainer design decision before code

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions