Skip to content

Long-running goals: zero mid-run progress events (milestone RFC not landed); operator blind for the entire run #5886

Description

@AronSwan

Environment

  • loopx v1.2.4 (milestone trigger source verified byte-identical in v1.3.1), orchestrating multi-agent research goals via turn run-once

What happened (measured)

A single research goal runs 40–85 minutes wall-clock (8-9 agent turns, measured across 5 production runs). During the entire run there are zero mid-run progress events an operator or dashboard could consume:

  • turn run-once emits nothing until final settle; drain only fires at the end.
  • periodic_report's bounded_segment_milestone trigger class exists in the RFC/design docs but is not implemented (source for the trigger is byte-identical between v1.2.4 and v1.3.1 — no code landed).
  • Practical consequence from our logs: the operator waited 83 minutes with no signal (run 2026-10-07), and we resorted to polling the filesystem for new artifacts.

Impact

  • status/todo list show coarse state, but "what is the agent doing / how far along" is invisible mid-run.
  • This is the single largest operability gap we've hit: audit after the fact is excellent (turn journal + receipts are great), visibility during the run is zero.

Suggestions (directions)

  • Land the milestone trigger (even a conservative "phase completed" event per turn would transform operability).
  • Or document an official wiring for orchestrators to emit progress from run-once (e.g. a --on-phase hook).
  • We currently plan to poll journal files as a workaround — happy to upstream that as a tiny loopx status --watch recipe if preferred.

Five fully-audited production runs behind these observations — happy to share timelines.

Activity

  1. mikamikasuki commented on Oct 8, 2026

    @mikamikasuki
    Contributor

    I’m taking a bounded, read-only CLI slice for this: add an opt-in watch mode to the existing Turn journal inspection command so operators can see persisted phase transitions while run-once is still executing.

    At canonical main 3ed5d6b, periodic_report does implement bounded_segment_milestone, but it is emitted for validated, durably settled stage closures; it does not surface ordinary phase progress during a long host Turn. turn run-once also renders one CLI payload after the synchronous executor and post-writeback hooks return. The existing turn inspect-journal command reads the canonical fenced journal without executing effects, so I’ll build on that boundary and show only safe journal status/phase fields, with no host output or session content.

    I’ll keep this optional and read-only, verify polling against concurrent journal checkpoints, and exit when the Turn reaches a terminal state. Please flag any conflict with the issue’s desired progress contract while I work.

  2. added 2 commits that reference this issue on Oct 8, 2026
    bb30c8a
    f4d9015
  3. mikamikasuki commented on Oct 8, 2026

    @mikamikasuki
    Contributor

    Opened PR #5998 with the bounded journal-watch implementation. It exposes persisted journal status and phase checkpoints through turn inspect-journal --watch, with allowlisted JSONL/Markdown output and no host/session content. The PR includes a concurrent-writer regression test and the English/Chinese integration recipe. It does not add transient host-step progress before a journal checkpoint is persisted.

  4. AronSwan commented on Oct 9, 2026

    @AronSwan
    ContributorAuthor

    From the filer side: no conflict — this matches what the issue was asking for. Two data points from our orchestrated runs that the design covers well:

    1. The 83-minute blind wait cited in the issue: persisted phase checkpoints would have shown the turn was alive and roughly where it was, which is all an operator needs to decide "keep waiting vs. investigate".
    2. The stuck-turn case we filed separately as turn run-once: quota_spend CAS failure leaves turn permanently in_progress (completed work unauditable) #5885 (quota CAS failure leaves the turn permanently in_progress): under "exit on terminal state", a stuck turn shows zero transitions and the watch never exits — which is itself the liveness signal an operator needs to intervene. The current surface gives nothing at all in that scenario.

    Read-only scope + journal_consistent gating + no host/session content is the right boundary. Following the review on #5998.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions