Repository navigation
feat: executor-verifier orchestration — executable acceptance, budget classes, model ladder, isolation - #21
Merged
Conversation
…olation
Five capabilities adapted from the-open-engine/zeroshot's executor–verifier
orchestration, rewritten for this harness's thesis: small local models, a fixed
runway, and gates that fail closed. Nothing is vendored — zeroshot is TS/Rust
and this is Go, so what carries over is the topology, not the code. Its central
claim (an independent LLM verifier) is deliberately NOT adopted: our gates ask
the filesystem, which is strictly stronger evidence than a second opinion from
the same weak model.
── Executable acceptance criteria ────────────────────────────────────────────
plan.Task.Acceptance is prose, and pkg/quality has always mined it for runnable
commands with a whitelist scan. That loses three things: WHICH condition failed,
whether a condition is blocking, and whether it was checked at all.
plan.Criterion{ID, Text, Verify, Priority} carries all three. quality.
VerifyCriteria runs each Verify through the same whitelist and sanitizer every
auto-run command already passes — a criterion cannot widen shell scope by one
character — and reports three verdicts, never two:
PASSED a command ran here and exited 0
FAILED it ran and did not; a failed `must` fails the task
UNVERIFIED nothing ran — no command, or one the whitelist refused
The third state is the point. The prose scan's silent third state is exactly how
"the harness did not check" becomes "the harness says it is fine". UNVERIFIED
never blocks, but it denies the reviewer fast path: disk evidence proves the
worker CHANGED something, never that the stated condition is now true.
Acceptance stays the compatibility surface — Normalize synthesizes it from
Criteria — so every consumer that predates the field is untouched, and a model
too small to emit criteria still produces a valid task.
── Budget classes ────────────────────────────────────────────────────────────
The composer already chose phases, team and skills; it never chose how much of
the run's budget a request is worth, so "rename a variable" could still walk
architect → explore → plan → split → execute → review → test. On a 7-32B model
that ceremony IS the failure: the runway is gone before the edit lands.
complexity (trivial|simple|standard|critical) × kind (inquiry|task|debug) buys a
Profile: optional phase breadth, wave budget, think passes, QA rounds, strict
reviewer. What it explicitly does NOT scale is reviewer COUNT — our reviewers
are LLM calls and our gates are not, so a higher class buys more determinism,
never more voices.
composer.Classify is a deterministic cheap tier that costs nothing and answers
the easy majority; it reports Confident only on a strong signal, and only
trivial/inquiry skip the composer LLM entirely. Critical still pays for it:
"confidently classified" is not "easy", and a change to auth or billing is
exactly where a model's judgment about specialists is worth its tokens.
Config is a ceiling for boolean gates — a class may switch one off, never on.
The single downgrade is RequireSmoke for a read-only inquiry, which writes
nothing and so could never satisfy it: a deadlock, not a safeguard.
── Role→model map and failure escalation ─────────────────────────────────────
The whole routing story was FastModel plus an isLightAgent classification: one
lever, two models. model_roles generalizes it to as many models as an operator
names. model_escalation answers a different question — this TASK keeps failing —
using state Task.AttemptLog was already recording and nothing read back.
A rung is a separately registered agent, identical to its base except the model
it is pinned to, so escalation stays out of the hot path and a task that never
escalates pays nothing. Every id-classification helper goes through BaseRoleID:
an escalated worker that stopped matching isCodingRole would keep running with
the wrong token budget, invisibly.
── Worktree isolation ────────────────────────────────────────────────────────
pkg/rewind already snapshots and restores per wave. Two things it cannot do:
abandon (one operation that cannot half-succeed, covering files nothing
predicted) and contain (an out-of-scope write lands somewhere the operator was
never using).
The seam is one line: point Config.Root at a throwaway git worktree. Every ws_*
tool already resolves against it, so the tool layer is isolated without knowing.
Config.StateDir pins .slmcode to the ORIGIN — derived from a throwaway worktree,
the memory layers, repair rules and bandit policy would be thrown away with it,
turning "it gets better at your repo" into "it starts from zero every run".
In-place stays the default permanently: plenty of workspaces are not git repos.
── Issue intake and pull-request delivery ────────────────────────────────────
`gh` rather than a Go client: already installed, already authenticated, no new
token to store or leak. An issue body is untrusted internet text that becomes
the run query, so it is framed as a report rather than pasted as instructions,
its harness markers are defused, and it is truncated so a long issue cannot
evict the repo map from the pack.
Delivery is the one thing here that leaves the machine, so it takes a flag AND a
confirmation showing branch, title and the exact file list; with no TTY it
refuses unless --yes. Only files the run's completed tasks named are staged,
never `git add -A`.
── Bugs found by running it ──────────────────────────────────────────────────
- Isolation did not isolate: openHarness builds the orchestrator, and its
workspace, from the original root before Config.Root is redirected. A live run
edited main.go in the origin. The orchestrator is now rebuilt after.
- Interrupting an isolated run leaked the worktree and its branch: teardown was
inline past six `return failf(...)` paths. Now deferred.
- vcs.BranchName("!!!") produced "slmcode/", which branchSafe accepts (slash is
legal) and git then rejects — a cosmetic input failing at the last step.
- The classifier missed inflected forms: "panics" did not match "panic".
- The review-time criteria insurance ran for roles the worker path skips, so a
tester task executed the project's suite twice, once per path, where the
per-report command cache could not see the duplication.
Supersedes the collapsible-panels approach: same complaint, structural fix
rather than a cap. The page is now four fixed zones in priority order, and the
ordering IS the design.
1. Command bar what you type and the button you press. One row on desktop,
wraps below. Never scrolls away.
2. Phase rail where the run is, in ~44px.
3. Run setup how it is configured, behind a disclosure.
4. Stream + rail everything left over, which is most of the screen.
── PhaseRail ────────────────────────────────────────────────────────────────
One horizontal track replaces four separate header panels — a metrics grid, a
percentage bar, a wrapped grid of phase chips and a "current stage" card — that
between them spent roughly a third of the viewport answering one question. It
answers it better too: position along the rail IS the progress bar, so the two
cannot disagree. The active phase scrolls itself into view.
Colour carries meaning and nothing else. Phases were tinted from a fifteen-entry
per-phase palette, which reads as decoration because it is — nobody holds
fifteen hues to fifteen phase names. The tint is now the phase's GROUP (five,
and groups are a real concept in pipeline.yaml); STATE is carried by fill and
weight.
── RunSetup ─────────────────────────────────────────────────────────────────
Composition, team, handoff and SLM-fit hints are genuinely useful, and useful at
a different moment than the log: you read them while deciding what to run and
stop the instant the run starts. Three disclosure states, not two:
preview collapsed — a deterministic GUESS from the query text
running collapsed — you are watching the log
runtime, finished open — this is what actually ran
A composition-read error renders OUTSIDE the disclosure. Hiding it behind a
collapsed panel means the one state the user must act on is the one they cannot
see.
── Screen adaptability ──────────────────────────────────────────────────────
The rail is a column at ≥1024px and a full-height overlay below it, seeded from
the breakpoint and reset when the viewport crosses it — an overlay defaulting to
open covered the console the user came for. The nav sidebar now forces its icon
rail below `lg`: at 224px wide it took 60% of a 375px screen, leaving the page
it navigates to to fight for the rest. The user's own collapse preference is
preserved, not overwritten, and applies again when there is room.
LiveFeedback gains a compact mode and docks to the bottom of the stream, where a
reply belongs — next to what you are replying to, not in a header card above it.
── The preview was presenting a guess as a decision ─────────────────────────
Reported: typing a query instantly produced a full "Planned pipeline" panel, and
for a Go repository it named python-worker and python-tester. It is now labelled
"Provisional pipeline · guess", collapsed, and says plainly that the composer
agent picks the real phases when the run starts. The wrong team itself is fixed
in the engine (see langpick.go).
The three surfaces a user meets these through: - cli.md gains the run flags (--issue, --isolate, --deliver and friends), what a budget class buys, and why the provisional pipeline shown while typing is a guess rather than a decision. - config.md gains model_roles, model_escalation and escalate_after. - concepts.md explains why acceptance is a contract rather than a paragraph, and what the third UNVERIFIED verdict is for.
…ng them This branch was cut before 0989f9f, which reworked the same Live page and added hooks/useUiState.ts and ui/ResizeHandle. The rebase kept this branch's structure — four zones, an extracted PhaseRail and RunSetup — but there is no reason for it to carry its own copies of infrastructure that now exists: - useMediaQuery is deleted; useUiState already exports one. - usePersistentState now backs the rail tab, the rail width and the RunSetup disclosure, so a layout the user arranged survives a reload. - useStickToBottom replaces an unconditional scrollIntoView on every event, which yanked the log back down mid-flush and made reading anything during a run impossible. It follows the stream only while the user is already at the bottom. - ResizeHandle restores the draggable, persisted side-panel width that the superseded version had. The stored width is clamped twice — by the handle and again in CSS — so a window narrowed after the fact can never leave the stream with no room. RunSetup's tests clear localStorage between cases: jsdom shares it across a file, so the tests that click the toggle otherwise leak an expanded panel into every test declared after them.
Left behind by an edit to the compact placeholder. The trailing-whitespace hook was the only failing check on this branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adapts the structural ideas from the-open-engine/zeroshot's executor–verifier orchestration, rewritten for this harness's thesis: small local models, a fixed runway, and gates that fail closed.
Nothing is vendored — zeroshot is TS/Rust, this is Go — so what carries over is the topology, not the code. Its central claim, an independent LLM verifier, is deliberately not adopted: our gates ask the filesystem, which is strictly stronger evidence than a second opinion from the same weak model. What was genuinely missing was everything around that verifier.
What's in it
1. Executable acceptance criteria
Task.Acceptanceis prose, andpkg/qualityhas always mined it for runnable commands with a whitelist scan. That loses three things: which condition failed, whether it's blocking, and whether it was checked at all.plan.Criterion{ID, Text, Verify, Priority}carries all three.quality.VerifyCriteriaruns eachVerifythrough the same whitelist and sanitizer every auto-run command already passes — a criterion cannot widen shell scope by one character — and reports three verdicts, never two:PASSEDFAILEDmustfails the taskUNVERIFIEDThe third state is the point. The prose scan's silent third state is exactly how "the harness did not check" becomes "the harness says it is fine".
UNVERIFIEDnever blocks, but it denies the reviewer fast path: disk evidence proves the worker changed something, never that the stated condition is now true.Acceptancestays the compatibility surface (Normalizesynthesizes it fromCriteria), so every consumer that predates the field is untouched and a model too small to emit criteria still produces a valid task.2. Budget classes
The composer already chose phases, team and skills; it never chose how much of the run's budget a request is worth. So "rename a variable" could still walk architect → explore → plan → split → execute → review → test. On a 7–32B model that ceremony is the failure — the runway is gone before the edit lands.
complexity(trivial|simple|standard|critical) ×kind(inquiry|task|debug) buys a profile: optional phase breadth, waves, think passes, QA rounds, strict reviewer.What it explicitly does not scale is reviewer count — our reviewers are LLM calls and our gates are not, so a higher class buys more determinism, never more voices.
composer.Classifyis a deterministic cheap tier that costs nothing. Onlytrivial/inquiryskip the composer LLM; critical still pays for it, because "confidently classified" is not "easy" and a change to auth or billing is exactly where a model's judgment about specialists is worth its tokens.Config is a ceiling for boolean gates — a class may switch one off, never on. The single downgrade is
RequireSmokefor a read-only inquiry, which writes nothing and so could never satisfy it: a deadlock, not a safeguard.3. Role→model map and failure escalation
The whole routing story was
fast_modelplus anisLightAgentclassification: one lever, two models.model_rolesgeneralizes it;model_escalationanswers a different question — this task keeps failing — using stateTask.AttemptLogwas already recording and nothing read back.A rung is a separately registered agent, identical to its base except the pinned model, so escalation stays out of the hot path and a task that never escalates pays nothing. Every id-classification helper routes through
BaseRoleID: an escalated worker that stopped matchingisCodingRolewould keep running with the wrong token budget, invisibly.4. Worktree isolation (
--isolate worktree, opt-in)pkg/rewindalready snapshots per wave. Two things it can't do: abandon (one operation that cannot half-succeed, covering files nothing predicted) and contain (an out-of-scope write lands somewhere the operator was never using).The seam is one line — point
Config.Rootat a throwaway worktree; everyws_*tool already resolves against it.Config.StateDirpins.slmcodeto the origin: derived from a throwaway worktree, the memory layers, repair rules and bandit policy would be thrown away with it, turning "it gets better at your repo" into "it starts from zero every run".In-place stays the default permanently — plenty of workspaces aren't git repos.
5. Issue intake and PR delivery
ghrather than a Go client: already installed, already authenticated, no new token to store or leak. An issue body is untrusted internet text that becomes the run query, so it's framed as a report rather than pasted as instructions, its harness markers are defused, and it's truncated so a long issue can't evict the repo map from the pack.Delivery is the one thing here that leaves the machine, so it takes a flag and a confirmation showing branch, title and the exact file list; with no TTY it refuses unless
--yes. Only files the run's completed tasks named are staged — nevergit add -A.6. Studio Live page
Four fixed zones in priority order, and the ordering is the design: command bar → phase rail (~44px) → run-setup disclosure → the stream, which now gets the space. One horizontal
PhaseRailreplaces four header panels that spent a third of the viewport answering one question; position along the rail is the progress bar, so the two can't disagree. Phase tint is keyed to group (5) rather than phase (15).The nav sidebar now forces its icon rail below
lg— at 224px it took 60% of a 375px screen.Not in it
context_role_budget(the "declarative context strategy" item). It's already wired config→packer and documented indocs/config.md. Building it would have been duplication, so I didn't.This branch was cut from
cbeab27, before0989f9f fix(studio): make the Live page fit its window and stop it flickeringlanded. Both address the same complaint. That commit made the existing panels collapsible with persisted state and capped the detail region at ~38vh; this branch restructures the page instead and extractsPhaseRail/RunSetup.The rebase kept this branch's structure, but the final commit deliberately adopts that commit's infrastructure rather than discarding it:
hooks/useMediaQuery.ts(mine) deleted —useUiStatealready exports oneusePersistentStatenow backs the rail tab, rail width and the disclosureuseStickToBottomreplaces an unconditionalscrollIntoViewthat yanked the log down every flushResizeHandlerestores the draggable, persisted side-panel width that would otherwise have been lostIf you'd rather keep
0989f9f's approach and drop the restructure, that's a reasonable call — the Live page changes are confined toweb/and the last two commits.Bugs found by running it, not by reading it
openHarnessbuilds the orchestrator — and its workspace — from the original root beforeConfig.Rootis redirected. A live run editedmain.goin the origin. Now rebuilt after.return failf(...)paths. Now deferred; Ctrl-C prints "isolated run discarded — your checkout is untouched".BranchName("!!!")→slmcode/branchSaferegex (slash is legal), then git rejects it — a cosmetic input failing at the very last step."panics"didn't match"panic".python-worker+python-testernext to "verify with go test ./... -count=1".queryLanguageSpecialistsbeatprojectLanguageSpecialistsunconditionally. Now the repository wins unless the inventory corroborates the query, which preserves the legitimate case (theweb/tree in a Go project). Seepkg/orchestrator/langpick.go.guess, saying plainly that the composer picks the real pipeline when the run starts.Verification
scripts/lint.shat its zero-issue baseline;-raceclean on every touched packageNot verified:
mkdocs build --strict—.venv-docshas a stale interpreter path from a moved directory (pre-existing, unrelated to this branch).