Skip to content

Frontier resets to ring0 whenever autocut advances 'next', so outer rings can be unreachable indefinitely — stable sat 3 weeks while next advanced 8 releases (follow-up to #1066) #1102

Description

@don-petry

Follow-up to #1066, and strictly worse than the dwell reset that issue describes. #1066 says a busy agent's dwell restarts each tick. This is about the frontier itself resetting, which means outer rings can be unreachable indefinitely rather than merely delayed.

The mechanism

cmd_promote advances exactly one ring per invocation: the frontier, defined in scripts/canary-rollout.sh (~L921) as the first channel in ring order whose commit differs from the candidate:

if [ "$ch" = "next" ] || [ "$c" = "$cand" ]; then :; else frontier="$ch"; break; fi

The candidate is whatever <agent>/v<M>-next currently points at. So every autocut that advances next makes ring0 differ again, and the frontier snaps back to ring0 — regardless of how far the outer rings had progressed.

Observed

dev-lead, 2026-09-08. ring0 and ring1 had both been walked up to 90dcc01c, leaving stable as the only lagging ring:

v139-next    27b0e8a3   <- autocut moved it
v139-ring0   90dcc01c   <- now differs from candidate => frontier resets HERE
v139-ring1   90dcc01c
v139-stable  40f6c232   <- the ring we actually needed to advance

promote reported gate=SOAKING for ring 'ring0' [next->ring0]. stable was three gated promotions away again, having been one away moments earlier.

Why this can be unbounded, not just slow

The cumulative dwell from next to stable for dev-lead is 4h + 8h + 12h = 24h (next->ring0 4h, ring0->ring1 8h, ring1->stable 12h). The canary schedule is cron: "33 */4 * * *" — every 4 hours. An agent whose watched paths change on most ticks therefore resets the frontier roughly every 4 hours, while needing 24 hours of uninterrupted progress to drain the pipeline.

For an active agent, stable is not merely delayed — it may be unreachable by normal promotion. dev-lead/v139-stable sat on 40f6c232 (2026-08-18) for three weeks while next advanced through eight releases (v139.4.0v139.11.0).

Second-order effect: it forces bad choices

Because the pipeline cannot drain, the only way to move stable is promote --override, repeatedly, walking each ring. That has two costs:

  1. It discards the soak evidence you were waiting for. On 2026-09-08 the fleet had 152 clean runs on 90dcc01c across three canary repos. Override-walking shipped 9e709367 — a commit with zero canary coverage — because each promote advances the target ring to the current candidate, not to the commit that was actually soaked.
  2. It normalises --override, which is meant to be exceptional.

Acceptance criteria

  • Advancing next must not reset progress already made on outer rings. Options: pin the frontier to a specific candidate for the duration of its rollout; allow multiple rings to advance per run when each one's own gate passes; or track per-ring target versions rather than deriving everything from the single current candidate.
  • It must be possible to promote a specific ring to a specific soaked release, without --override and without walking the whole chain. Note cmd_rollback looks like this but is unusable for v-scoped channels — see the sibling issue on its major-scope bug.
  • Cumulative dwell to stable must be reachable given the schedule interval, or the mismatch must be surfaced explicitly (e.g. "pipeline cannot drain at current cut cadence") rather than silently never completing.
  • Regression test on the observed shape: rings walked to a candidate, then next advanced — outer-ring progress must not be discarded.

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reports

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions