fix(fusion): make the workers do the work - #406
Merged
Conversation
The old intro described a mode that no longer exists. It said 'a cloud one that thinks, and local ones that do the bulk' — but either seat takes either kind now — and it told the operator to set the worker count in a control that was removed, because the count is the machine's ceiling and the orchestrator's per-fan-out call, not a setting. The new text says what the shape is, names the two legs it actually resolved, and then does the one thing the old one never did: it invites the pairing. Cloud planning with local workers, a local model planning while cloud workers execute, two cloud models, a big local model directing a small one — each is a different instrument, and moving that line is the point of the mode. The mark is a tree now: the model that plans on top, the ones that do underneath. No glyph encodes where a model runs, because that stopped being a property of the seat.
A real session showed the first lock was a latch, not a rule. The fan-out came back with one task `needs_orchestrator` and two `cancelled`; the gate had opened on the mere completion of a `fusion.delegate` call, so the orchestrator wrote fifteen files and ran five commands itself. `delegate` returns `ok` even when every worker failed — by design, partial results are the value of a fan-out — so "something was attempted" was never the right key. The rule now: a mutation is allowed only while the turn holds work a worker physically could not do, i.e. a task returned `needs_orchestrator` because it hit an approval a worker cannot request. That is the one thing the orchestrator has and the workers do not: a person at the other end. A first draft, a rewrite, a part that timed out — all of it goes back out as another fan-out. The refusal says so, and names what a re-delegation brief should carry, because the model's wrong conclusion last time was "the tool is broken, I will do it myself". The ledger reads the delegate result's own per-task statuses rather than taking a flag from the caller, so it cannot drift from the result the model is reading in the same step. An MCP tool no longer gates on its own `readonly` claim. That flag is the server's `readOnlyHint` / `destructiveHint` — third-party wire data the adapter itself says cannot be trusted for batch safety — so a server could have opted out of the rule by shipping one field. MCP tools are treated as mutating, which fails closed. Three defects from the same session's trace, fixed alongside: **A separator slip no longer ends the turn.** The model emitted `fusion_delegate`, one underscore short of the `__` escape qualified names travel under, and the step executor's membership check throws for the whole batch — a thirty-minute turn ended with `reason=failed`, twice in a row, over a separator. `resolveToolName` tries the forms a model actually confuses: single underscores as dots, a name that arrived still escaped, and case. Deliberately not fuzzy — `os.fs.write` must never resolve to `os.fs.trash` because they are one edit apart — so anything else still throws as before. **A fan-out of one.** `MIN_SLOT_CONTEXT` was 16,384, borrowed from `MIN_AUTO_CONTEXT` on the assumption that a worker needs what a chat session needs. A 12B model whose auto context lands at ~16-24k therefore divided into exactly one slot and every fan-out ran sequentially. Measured instead: the observed worker's stable prefix was 7,388 tokens, so 8k is the floor with room to finish a tool call. A GPU launch also never falls below two slots — a fan-out of one is the orchestrator queueing behind itself, all of the delegation overhead and none of the parallelism. **A ten-minute worker cap.** Two of three tasks were cancelled at exactly 600s: a 12B model writing six modules and their tests, queued behind another worker on that single slot. Now twenty minutes. A stuck worker still ends; a slow one now finishes. The `### fusion` guidance moved with the rule: it states the refusal, and says rework goes back out until the orchestrator would sign off on it.
… builds The workers wrote nothing. That is the whole bug, and every symptom the operator reported falls out of it. A worker turn has no operator at the other end — the TUI shows the parent session — so `worker-runner` installs a refuse policy and every approval-gated call dies on it. At approval level 1 that is EVERY write. The trace: six tasks, zero files, four returning `needs_orchestrator` with replies saying "the orchestrator must run these steps", two burning the 20-minute timeout retrying blocked writes. The orchestrator then did exactly what four blocked workers told it to do — and the gate's handed-up escape waved all thirteen writes through, because at level 1 that escape is not an exception, it is the normal case. So the orchestrator was not ignoring the harness. The harness had made it the only party able to act. **Workers can now act.** `fusion.delegate` asks the operator once, before any worker starts, naming the task titles and the directory the fan-out will write in. The category `fusion_fanout` sits at level 4 and is deliberately NOT grantable: a session grant would hand every later fan-out — different tasks, different directory — the same authority silently, and seeing this task list is the entire point of asking. Approving it grants each worker session those directories through a new `FanoutScopeRegistry`, consulted by `requireFsApproval` — the single funnel every fs and git mutation already passes through with absolute paths. The refuse policy stays for everything outside the scope, so straying comes back as a task to re-delegate rather than a dead worker. The agent's own trust surface is excluded: a scope that happens to contain `config.json` or `.env` must not become the way around `trust_config` never being grantable. The scope is derived from the brief — `task.files` first, then paths written into the instructions, because the orchestrator that exposed all of this put every path in prose and left `files` empty. The working directory floors it; `/` and `$HOME` are refused outright, since one answer authorises several workers to write unattended. **The orchestrator no longer builds at all.** The handed-up escape is gone and there is no circumstance in which it may mutate. Its refusal now tells it to re-send the task with the paths named, so the next fan-out prompt can authorise the directory a worker was blocked on. Also: the worker brief now says `os.fs.write` creates its own parent directories. Three of the six workers burned steps on `mkdir -p`, which needs an approval they cannot get and was never necessary. The integration test is the proof. The same level-1 fan-out whose third task used to come back `needs_orchestrator` with zero writes now comes back `ok`, file written, no errors — from one operator answer.
…⇄ that swaps the legs Three things the first clean end-to-end run asked for. **Workers can run commands.** The fan-out approval already authorised writing in a directory the operator saw; it now authorises running commands there too — `cwd` inside the scope, with the shell guard's own hardline `block` verdicts still firing above it. Half the tasks worth delegating end in "run the build, run the tests", and a worker that cannot was handing them up to an orchestrator that is refused every mutating tool: a step that never happens. The brief says so, the prompt says so, and the refusal text no longer tells a worker that "the orchestrator must run this". The command line itself is free text and cannot be scoped, so the directory is the whole of the promise — which is why the prompt reads "may write files and run commands in" rather than anything broader. **One question per turn, not one per fan-out.** A turn is one job: the orchestrator that reviews and re-delegates ran five fan-outs to build one library and asked the operator five times. Answering the same question five times is not consent, it is attrition. The answer now stands for the rest of the turn while later fan-outs stay inside the directories it named; one reaching somewhere new asks again, and `agent-loop` drops the standing answer at the start of every turn, so the authority never outlives the job it was given for. **⇄ swaps the legs.** The glyph between the two model names was punctuation inside one label. It is now its own button: one click puts the local model in charge and sends the cloud one to the workers, pins, active provider and model labels together in a single config write. `/runmode swap` is the same move from the keyboard, because a control only a mouse can reach is not a control in a terminal app. The two halves come apart with it — the left one opens the orchestrator's model switch, the right one the worker leg's. Also, from the same run: - A `tasks` argument that arrived as JSON *text* now parses instead of dying on "tasks must be an array". A 12B orchestrator on the text-JSON transport quotes the array often enough to kill a run, and refusing it taught the model nothing it could act on. - The worker timeout goes to 45 minutes. At 1200s a worker that had already written four of its six files was cancelled mid-run and lost the lot; a stuck worker still ends, and the asymmetry is the argument.
plombeer31
changed the title
…rovider id Found by clicking the new ⇄ in a real TUI: the strip came back reading `local-llama ⇄ anthropic/claude-opus-5`. A llama-server provider row carries no `chatModel`, so the orchestrator label fell through to the provider id — the exact lie the worker half was fixed for, on the other half of the same pair. Mirrors the worker branch: a local leg is named by the model the managed daemon actually serves. It matters more now than it did, because the two legs are one click apart rather than two trips through two switches.
…e tools
With the legs swapped so the local model orchestrates, the cloud workers
never ran. The fan-out was not failing; it was never being called.
The GBNF grammar is the local model's ENTIRE vocabulary of tool names —
the constrained sampler cannot produce a name the `tool-name` rule does
not list, whatever the catalog says. `fusion.delegate` was not in it. The
trace of the operator's own run shows the consequence exactly:
"The user said yes to dispatching the 6 worker briefs via
fusion.delegate. I need to call fusion.delegate on its own…"
</think>
[{"tool":"finish","args":{"summary":"Dispatched 6 independent
worker briefs via fusion.delegate…"}}]
It reasons its way to the right call, cannot spell it, and is pushed into
`finish` — reporting a fan-out that never happened. Verified against the
operator's own llama-server (qwen-3.8-27b), told to emit nothing but a
`fusion.delegate` call:
old grammar → [{"tool":"tool.view","args":{"path":"/app/README.md"}}]
new grammar → [{"tool":"fusion.delegate","args":{"tasks":[…2 tasks…]}}]
This stayed invisible for as long as the orchestrator was always a cloud
provider: those carry a native `tools` payload and no grammar at all. The
first local orchestrator found it immediately.
**The git write tools had the same hole.** `git.status/log/diff/show/
blame/branch` were listed, the nine write verbs from the local-first git
work were not — a local model could inspect a repository and never commit
to one, with nothing in the logs to say why. Same one-line rule, same fix:
`git.init/add/commit/checkout/clone/remote/fetch/pull/push`. Also verified
live: `[{"tool":"os.git.commit","args":{"message":"wip"}}]`.
Both are unconditional, as `tasks.*` already is: which tools exist is the
registry's business, not the sampler's. A fusion worker is grammar-
constrained too, and `fusion.delegate` refuses a call from inside a worker
session on its own — the same shape the grammar's `finish` already relies
on.
The tests pin the class of bug, not just these names: a descriptor the
grammar cannot emit is a tool local models cannot call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Make fusion actually delegate
Fusion is meant to run in three beats: the orchestrator plans and briefs,
the workers do the job, the orchestrator reviews and sends back what is
weak. Beat 2 never happened — two consecutive sessions ended with the
cloud orchestrator writing every file itself, once overwriting a file a
worker had just written.
The workers wrote nothing, and the harness is why. A worker turn has
no operator at the other end, so
worker-runnerinstalls a refusepolicy; at approval level 1 that refuses every write. Six tasks, zero
files, four returning
needs_orchestratorwith replies saying "theorchestrator must run these steps" — and the gate's handed-up escape
then waved all thirteen of its writes through. The orchestrator was not
ignoring the harness; the harness had made it the only party able to act.
What this branch does
One approval per fan-out, and workers that can act.
fusion.delegateasks the operator once, before any worker starts, naming the task titles
and the directory the fan-out will write in. New category
fusion_fanout, level 4, deliberately not grantable — seeing thetask list is the entire point of asking. Approving it grants each worker
session that directory through a new
FanoutScopeRegistry, consulted byrequireFsApproval, the single funnel every fs and git mutation alreadypasses through with absolute paths.
The orchestrator never builds. Every mutating tool is refused for the
whole orchestrator turn — read, delegate, reply, and nothing else.
Rework goes back out as another fan-out, so
needs_orchestratormeans"send it again with the path named", not "take over".
Workers may run commands too. Half the tasks worth delegating end in
"run the build, run the tests". A command whose
cwdis inside theapproved directory needs no second permission; the shell guard's
hardline blocks still fire above it.
One question per turn, not one per fan-out. A turn is one job: the
orchestrator that reviews and re-delegates ran five fan-outs to build one
library and asked five times. The answer now stands for the rest of the
turn while later fan-outs stay inside the directories it named, and is
dropped at the start of every turn.
⇄ swaps the legs. The glyph between the two model names is now a
button: one click puts the local model in charge and sends the cloud one
to the workers, pins and active provider and labels in one config write.
/runmode swapis the same move from the keyboard.Plus, from the same run: a
tasksargument that arrived as JSON textparses instead of dying on "tasks must be an array", and the worker
timeout goes to 45 minutes — at 1200s a worker that had written four of
its six files was cancelled mid-run and lost the lot.
Verified
fusion-r3on the built binary: orchestratoros.fs.write= 0, 9 gaterefusals, workers wrote all 13 files across 5 fan-outs with review in
between. Unit and integration coverage for the scope derivation, the
turn-scoped grant, the gate, shell-within-scope, the tasks-as-string
parse and the swap.
Suite: 9429 passing. One pre-existing failure on
main(
src/sidecar/send-message-concurrency.test.ts— a third turnm3leaks past the per-session controller); it fails identically on
origin/mainand is unrelated to this branch.