Repository navigation
Conversation
E2E tests did not runE2E tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer with write access must add the See E2E testing guide for details. |
Code Review by Qodo🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)
Great, no issues found!Qodo reviewed your code and found no material issues that require reviewTip of the day💡 Did you know, you can route each severity your way: inline, summary, both, or drop |
PR Summary by QodoDocument host-side, per-task runtime and model routing policy
AI Description
Diagram
High-Level Assessment
Files changed (6)
|
Every runtime/model/effort/persona choice is static per agent (ADR 0091, ADR 0104); CEL overlays cannot set model or effort and the normalized event carries no diff size, so a one-line docs change and a 2,000-line CI refactor get the same opus-at-high review. fullsend-ai#5777, fullsend-ai#6891, fullsend-ai#2842 and agents#497 ask for a cheaper model or effort on trivially scoped changes, agents#535 for a deeper review on security signals, fullsend-ai#6560 for a cost preview, and fullsend-ai#6964 / agents#1225 record the retry-on-unavailable incident class that a pre-sandbox served-model check would end. ADR 0112 decides that `fullsend run` resolves the four fields an `agents:` entry holds per task, on the host, before the sandbox exists, from a `routing:` block in config.yaml: rules keyed on system-derived task facts only (never PR text), a precedence slot between the per-run overrides and the static entry, guards (allowed set, runtime capability matrix, never-downgrade set kept as fullsend-ai#6527 wrote it, persona floors, served-model preflight), effort before model, every choice echoed, a shadow mode, and learning that lands only as a reviewed policy change. The two rejected options are per-prompt routing inside the sandbox (the fullsend-ai#6361 draft) and routing in an LLM gateway, which stays an optional provider. Living documents: a Decided line under Agent Runtime in architecture.md and annotations on the open questions in adaptive-agent-selection, cross-run-memory and operational-observability. No code change. Assisted-by: Claude Claude-Session: https://claude.ai/code/session_01BCAjgeumZBcuaRSVvsbAkY Signed-off-by: Wayne Sun <gsun@redhat.com>
Rules cannot read a diff; a short read-only run on the fast tier can. Record the advisor as a follow-on decision and fix the one constraint this ADR owns for it: an assessment read from author-controlled content may fire an escalation rule alone but may satisfy a downgrade rule only together with a system-derived fact, and it never widens the allowed set or crosses the never-downgrade floor. Assisted-by: Claude Claude-Session: https://claude.ai/code/session_01BCAjgeumZBcuaRSVvsbAkY Signed-off-by: Wayne Sun <gsun@redhat.com>
Define downgrade classification from an ordered allowed_models list and the effort scale, name review in the never-downgrade defaults, merge routed subagents per persona key as ADR 0104 does, pin fail-closed defaults and the preflight miss path, redact author-controlled event and linked-issue fields from routing inputs, and drop the undefined persona min_tier floor. Drop the cross-run-memory annotation to stay within relates_to, and link ADR 0091's precedence chain to 0112. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tim Pouyer <tim.pouyer@gmail.com>
…atch Classify downgrades by an ordinal catalog tier, define persona min_tier as a floor on routed models only, and reject routed subagents a runtime cannot serve, applying subagents.default on Claude Code through CLAUDE_CODE_SUBAGENT_MODEL. Give rules ADR 0098's redacted entity and nullable event, source attempt count from handled-state evidence, tie trusted labels to 0098's label provenance, fail closed on actorless poll runs, and join outcomes on the entity's stable identity. Gate enforce on replay, pin advisor containment, require gateway providers to report the serving model, cite escalate-only prompt routing as prior art, and open the question of a decision model (Jev, Laya) as the advisor's classifier. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tim Pouyer <tim.pouyer@gmail.com>
1f76d83 to
3a5467d
Compare
Site previewPreview: https://eca528eb-site.fullsend-ai.workers.dev Commit: |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
/fs-review |
|
🤖 Finished Review · ✅ Success · Started 7:31 PM UTC · Completed 7:57 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $7.57 |
|
Risk Assessment: moderate (2/5) DetailsDocs-only ADR update with no protected paths or code changes, elevated to moderate risk by high commit churn, regression history, and co-change coupling on touched architecture documentation. |
ReviewFindingsHigh
Medium
Low
This PR is docs-only (a new Architecture Decision Record plus four small cross-reference edits — no production code). Per this repo's ADR convention, an accepted ADR is treated as effectively frozen afterward (substantial rewrites require a new superseding ADR), so the internal-consistency and cross-ADR-consistency gaps above are worth resolving in this PR rather than in a follow-up, particularly the Next steps:
|
| **Decided (implementation):** | ||
|
|
||
| - The `fullsend run` runner delegates in-sandbox agent execution to a `runtime.Runtime` interface; production orgs default to Claude Code, with [pi](https://github.com/earendil-works/pi) available as an opt-in second runtime (`runtime: pi`, Claude-on-Vertex through the same WIF credential path) and [codex](https://github.com/openai/codex) as a third (`runtime: codex`, OpenAI-only through a custom model provider whose bearer token comes from a runner-seeded file, with the sandbox tool hooks behind a translating adapter — [ADR 0099](ADRs/0099-codex-agent-runtime.md) and [ADR 0100](ADRs/0100-codex-sandbox-hooks.md)). Runtime selection is configured per repo with `runtime:` in `.fullsend/config.yaml` (per-agent `runtime`/`model`/`effort`/`subagents` on the agent's `agents:` entry sit above it and below the `--runtime`/`--model`/`--effort` flags and `FULLSEND_*` variables, [ADR 0091](ADRs/0091-per-agent-runtime-model-effort.md)) and resolved via `runtime.ResolveForAgent()`. Test-only runtimes — **dummy** (scripted operations) and **dummy-playback** (playlist-based replay of canned results) — execute in the real OpenShell sandbox for behaviour tests without inference. Bootstrap uses a portable `BootstrapInput` interface with optional extensions such as `SandboxHooksBootstrap` for the runtime-neutral sandbox tool hooks ([ADR 0090](ADRs/0090-runtime-neutral-sandbox-hooks-contract.md)); runtimes declare further capabilities through small optional interfaces (`DebugLogNamer`, `ContextBridger`) rather than `Name()` checks in the runner. Transcript and debug artifact handling use a separate `TranscriptHandler` interface. See [runtimes.md](runtimes.md) for the per-runtime security feature matrix required when adding a new backend. | ||
| - Runtime, model, effort and persona models are routed **per task** by a `routing:` policy in `.fullsend/config.yaml`, resolved by `fullsend run` on the host before the sandbox exists, from system-derived task facts only (forge change stats and paths, actor kind, allowlisted labels, attempt count — never PR text); a rule sits below the per-run flags and variables and above the agent's `agents:` entry, is bounded by `allowed_models`, a never-downgrade set, persona `min_tier` floors on routed models and a served-model preflight, and is echoed in the plan block, `metrics.json`, the root span and the status comment; `shadow` mode records the decision without acting on it, and learning lands only as a reviewed policy change ([ADR 0112](ADRs/0112-task-grain-routing-policy-on-the-host.md)). |
There was a problem hiding this comment.
[medium] api-contract
docs/architecture.md's frontmatter and intro explicitly describe the document as 'the current architectural truth, kept in sync with accepted ADRs' and state that ADRs 'crystallize' into this document as the source of present-tense fact. The new bullet added by this PR sits under 'Decided (implementation):' — alongside bullets describing shipped code (runtime.Runtime, ResolveForAgent()) — and states in the present tense that runtime/model/effort/persona selection 'are routed per task' by a routing: policy. ADR 0112 is a docs-only decision record with no implementation; nothing in .fullsend/config.yaml or fullsend run implements routing: yet. As placed, the bullet misrepresents an accepted-but-unimplemented decision as current production behavior.
Suggested fix: Rephrase the bullet to distinguish decision from implementation, e.g. 'Accepted in ADR 0112 (not yet implemented): runtime, model, effort and persona models will be routed per task ...', or move it to a clearly-marked 'Decided, not yet implemented' subsection until routing: actually lands in code.
There was a problem hiding this comment.
Fixed in 3e2289e. The bullet now reads "will be routed per task … (decided in ADR 0112, not yet implemented)". I've kept it under "Decided (implementation):" because architecture.md has no separate list for decisions that are accepted but not yet built, and the bullet says so inline. When routing: ships, the future tense and the caveat both come out.
Require trusted downgrade labels to come from a human non-author with write, and default trusted_labels and allowed_models to empty. Fail the run on a malformed routing block. Forbid routed persona tombstones and floor never-downgrade personas at their static tier unless they opt in. Classify escalation versus downgrade on the fully resolved result, keep escalations from being undercut by later downgrade matches, and recast effort-before-model as guidance. Describe routing's CEL environment as its own allowlist, make task.actor null on poll runs, narrow the no-retry claim, make the advisor constraint checkable, and prefix tier uses. Mark the architecture.md line as decided but not implemented. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tim Pouyer <tim.pouyer@gmail.com>
waynesun09
left a comment
There was a problem hiding this comment.
Review — routing policy ADR
Independent review of the ADR text and the example config against main @ afa135d45, plus a drift check on every cited reference. Complements the fleet review above rather than repeating it: everything below is a finding that review did not raise.
The decision is sound and eight of the nine findings from #7251 are genuinely closed — verified against the ADR text rather than the summary table. The guards are computed rather than declared, min_tier is defined, subagents merges per key, the preflight fallback is specified, and the fail-closed defaults are real. Number 0112 is free on main and claimed by no other open PR (re-scanned all open PRs; 38 touch docs/ADRs/). All eight sample CEL expressions compile to bool against the repo's own cel-go and all three sample YAML blocks parse. The ADR 0091 edit is a permitted minor annotation, not a rewrite.
Two findings I would treat as blocking, both small edits:
- Decision §4 states three runtime-state snapshots as decision clauses. One is contradicted by ADR 0104 and by PR #7809, open right now, which registers codex personas; another is factually false today. An Accepted ADR whose Decision is wrong on merge day needs a superseding ADR to fix a sentence that was never the decision. Inline at the two lines.
- The PR body and the ADR specify opposite behaviour for a malformed
routing:block — the body says the static choice runs, the ADR says the run fails. The ADR is the correct, fail-closed version; the body needs the fix. Flagging because the body is what a reviewer approves against. (The fleet review reads this as an internal ADR contradiction; I read:109-111as self-consistent and the body as the defect.)
Example-config defects (not inline — they live in the PR description)
docs-only-review, the flagship downgrade, is rejected by the ADR's own persona floor.:160-161floors anever_downgradeagent's personas at their static model's tier unless the persona file declares a lowermin_tier. The sample moveschallengerandcorrectnessfromxai/grok-4.6togoogle-vertex/gemini-3.8-flashwith no opt-in shown, so as written it fails validation. Either show themin_tieropt-in in the persona sample or pick personas that are not floored.retry-escalateis labelled an escalation but classifies as a downgrade under §4's own test.:140-141makes a rule an escalation only when every resolved field is equal to or above static and at least one is above;:144-145makes "anything else, including a runtime change or a same-tier model swap" a downgrade. Forfix, whose static model issonnet,{model: opus, effort: high}raises the model but the effort comparison against an unset static value is undefined (see also C4). Worth making the sample unambiguous, since it is the one readers will copy.bad-title-keyuses a field shape this repo's CEL does not have.eventisnormevent.Event.ToMap()— six top-level fields, nopull_request, no title anywhere. The sample is right that PR titles must be unreachable, but it demonstrates it with a path that would fail for the wrong reason. Suggest keying the example on a field that does exist in the redactedevent.
Drift check
Everything cited resolves correctly with one exception: #6527 is CLOSED (2026-08-23), so :204's "a later ADR that amends #6527" names a dead artifact. Inline below.
Worth recording that ADR 0098 is entity-first harness evaluation on main today (#6956, merged 2026-09-15) and 0106 is serialize-agent-runs (#6909, merged 2026-09-16) — I went in expecting the reverse and was wrong. The citation is correct and needs no change.
Length
248 lines against a main median of 131 (p90 283, n=103) — within practice but roughly double the median, and the skill's own targets are exceeded on Context (36 lines) and Consequences (61). The Jev/Laya block at :228-248 is the largest single win and reads as the open-ended exploration skills/writing-adrs/SKILL.md says belongs in a problem doc. Suggest collapsing it to one Deferred bullet and moving the substance to adaptive-agent-selection.md; happy to supply that text.
Agent-assisted review. Every source claim below was read at main @ afa135d45 or in the linked PR; file:line references are quoted so they can be checked.
| entry still resolves by ADR 0104's order. A runtime change must keep the | ||
| agent's needs (fallback chain on Claude only), and routed `subagents` must | ||
| be servable on the resolved runtime: on codex, which has no persona | ||
| roster, any `subagents` key is rejected; on Claude Code, |
There was a problem hiding this comment.
BLOCKING — decision-vs-state. "on codex, which has no persona roster, any subagents key is rejected"
This states current runtime state as a decision clause, and it is already contradicted by two sources:
- ADR 0104:110-112 records the opposite status: "Consuming the same map on codex (
subagents.default, OpenAI ids) is planned, not implemented here." ADR 0112 turns another ADR's "planned" into a standing prohibition without superseding it. - PR feat(#6970)!: support native Codex review and retro children #7809 is open right now —
feat(#6970)!: support native Codex review and retro children. Its own Changes section: "Register discovered personas plus generic and Explore roles. Resolve child models from an explicit persona override, thenFULLSEND_CODEX_SUBAGENT_MODEL, thensubagents.default…" If it merges, this Decision sentence is false and correcting it needs a superseding ADR.
Today's actual state is only internal/runtime/codex_bootstrap.go:311-320 (codexNoSubagentNote, a prompt note that the sub-agent tool is absent) — a fact, not a decision.
The guard's intent is right and belongs in the Decision; the per-runtime roster inventory is a snapshot that changed while this PR was open. docs/contributing/adrs.md lets a later PR annotate a note but not rewrite a Decision.
Suggested edit — keep the runtime-agnostic rule here:
A runtime change must keep the agent's capability needs, and every routed
subagentskey must be servable by the resolved runtime's persona mechanism: a key the runtime cannot serve fails validation before the sandbox exists, naming the runtime and the key. Each runtime's persona mechanism and the models it can carry are the runtime's to define (ADR 0104).
and move the inventory to a dated Consequences note ("Runtime state at acceptance (2026-09-11; annotate, do not rewrite)") naming pi as the only runtime registering personas today, codex as tracked by #6970, and Claude Code as pending ADR 0104's second half. When #7809 lands, codex becomes routable under the rule above with no change to this ADR.
There was a problem hiding this comment.
Agreed, thanks. Addressed in cd903f6. I adopted the runtime-agnostic rule you suggested (every routed subagents key must be servable by the resolved runtime's persona mechanism, or validation fails), and the same commit trims the ADR to the decision alone, so it no longer names any runtime. The servability rule and a dated note on current runtime status (pi registers personas; codex is tracked by #6970 and #7809) are left to the implementation. If #7809 merges, codex personas become routable with no ADR change.
| may declare `min_tier:`, a floor on *routed* persona models only: a rule | ||
| cannot place the persona below it, while a static `subagents.<persona>` | ||
| entry still resolves by ADR 0104's order. A runtime change must keep the | ||
| agent's needs (fallback chain on Claude only), and routed `subagents` must |
There was a problem hiding this comment.
BLOCKING — factually false on main. "(fallback chain on Claude only)"
pi has a fallback chain today. internal/runtime/pi_run.go:1036-1044 documents it verbatim — "When the requested model is an alias and FallbackModels are set, Run attempts each model in the chain until one succeeds… A Vertex 404 … triggers the next fallback" — implemented at pi_run.go:1064 via piFallbackChain(effectiveModel, params.FallbackModels, params.ModelAliases). Only the stale comment at internal/runtime/runtime.go:66 still reads Claude-only.
As written, this parenthetical is a guard that would wrongly block a Claude→pi routed runtime change, on a capability pi actually has.
Same at :168: "Claude's existing --fallback-model chain is unchanged" → suggest "the runtimes' existing fallback chains are unchanged".
Suggested edit: drop the parenthetical entirely — "A runtime change must keep the agent's capability needs" carries the intent without naming a per-runtime inventory that is already out of date.
There was a problem hiding this comment.
Fixed in cd903f6. The parenthetical is gone, and the ADR now refers only to each model's existing fallback chain, with no per-runtime list. It also narrows the preflight so it doesn't fight those chains: a static model that isn't served fails the run only when nothing in its existing fallback chain is served, whether that's Claude's --fallback-model or pi's piFallbackChain. The preflight runs only when routing.mode isn't off, so repos that haven't opted in see no new failure mode.
| agent's needs (fallback chain on Claude only), and routed `subagents` must | ||
| be servable on the resolved runtime: on codex, which has no persona | ||
| roster, any `subagents` key is rejected; on Claude Code, | ||
| `subagents.default` is applied through `CLAUDE_CODE_SUBAGENT_MODEL` (never |
There was a problem hiding this comment.
HIGH — mechanism is real but inert for every fullsend review persona. "subagents.default is applied through CLAUDE_CODE_SUBAGENT_MODEL"
The variable exists and the ordering stated is right, but per Claude Code's documented sub-agent resolution — per-invocation model argument › frontmatter model: › CLAUDE_CODE_SUBAGENT_MODEL › parent — it only reaches children that name no persona and receive no explicit model.
All nine pr-review personas in fullsend-ai/agents carry a frontmatter model: (verified today), so every one of them outranks it. The variable also appears zero times in the fullsend tree. So on Claude Code this clause routes nothing that fullsend currently dispatches.
Not blocking on its own, but it should not read as a working mechanism. Suggest stating the limitation where the mechanism is named, or folding it into the dated runtime-state note proposed above.
Separately, CLAUDE_CODE_SUBAGENT_MODEL_FORCE (:155) is not in the official docs — I could only find it in secondary sources describing a boolean that forces one model onto subagents. Worth a primary citation or removal.
There was a problem hiding this comment.
Agreed. The ADR no longer names the variable (cd903f6). The runtime details are left to implementation, and I've recorded the limitation for that work: on Claude Code only subagents.default is routable, it reaches only children with no persona and no frontmatter model:, and a per-invocation model can override it. So the echo records the routed value, not necessarily the one that served. _FORCE is gone. The Claude Code 2.1.251 boundary is still unverified. If you know the release that introduced it, I'll cite it.
| calibrated probability for each label of a closed set, in tens to hundreds | ||
| of milliseconds by their vendors' figures, so the `scope`, `risk` and | ||
| `domains` enums map onto them directly, a confidence floor reads a real | ||
| probability, and the ledger records the distribution in place of a reason. |
There was a problem hiding this comment.
HIGH — "a confidence floor reads a real probability" is true only per backend, and is the one claim that does not survive making this pluggable.
confidence is not a portable quantity. Three backends that all serve the same POST /v1/systemone request body return three mathematically different numbers under that key:
| Backend | What confidence is |
|---|---|
| Jev | (n·p_max − 1)/(n − 1) — p_max rescaled so uniform → 0 (third-party reading; TypeSafe publishes no formula) |
| djev-run | raw p_max — no rescaling, no calibration |
| Laya | 1 − normalised entropy |
Laya states the consequence outright: "A threshold carried over from Jev does not transfer." Worked example, and it is not academic — magpie gates at 0.4: meaningful under Jev's scaling, but under raw p_max a two-way question at chance already scores 0.5, so the floor can never fire. Silently: no error, no log line, the abstention guard just stops existing.
Two consequences for this ADR, both cheap because it is deferring the choice anyway:
- The provider contract should carry the full
probabilitiesdistribution (all backends return it) and fullsend should compute the gate itself. A rule must never gate on a vendor'sconfidencefield. - Calibration does not transfer either, so it is not the differentiator that justifies a backend. Laya's own benchmark shows miscalibration reversing direction on a routing task. The threshold has to be fitted on fullsend's own labelled runs regardless of backend — which §7's ledger, shadow mode and replay already provide. That reframes calibration from a vendor property into a workflow this ADR owns.
Suggest softening :240 to "the ledger records the distribution, and fullsend computes the abstention gate from it" and adding the fitting loop to the open question. Full survey available if useful.
There was a problem hiding this comment.
Thanks, this was a good catch. The advisor is deferred, so I've kept the detail out of the ADR. The adaptive-agent-selection.md open question now says only that a decision model returns a probability per label, so fullsend could set its own abstention threshold from that distribution. I've recorded your two points for the advisor's implementation: never gate on a vendor's confidence field, and fit the threshold on fullsend's own labelled runs through shadow mode and replay.
| a rule is an *escalation* only when every resolved field is equal to or | ||
| above the static choice and at least one is above it, with effort ordered | ||
| `low` < `medium` < `high` < `xhigh` < `max` and models by the ordinal | ||
| model tier (economy < standard < premium) that each catalog entry carries |
There was a problem hiding this comment.
MEDIUM — present tense for fields that do not exist. "the ordinal model tier (economy < standard < premium) that each catalog entry carries beside its vendor"
ModelsConfig is Aliases only (internal/config/config.go:399-406); there are zero hits for a tier field anywhere in the tree. The PR's own "Notes for review" says so — "the ADR commits the implementation to adding tier and vendor to catalog entries, which don't exist today" — but the Decision text asserts them as current.
Since the whole downgrade/escalation classification rests on this ordinal, the reader should not have to reach the Notes section to learn it is not there yet.
Suggested edit: "…by an ordinal model tier that each catalog entry will carry beside its vendor (neither field exists today; adding them is part of implementing this ADR)."
There was a problem hiding this comment.
Fixed in cd903f6. The Decision now classifies models "by a catalog model tier", and Consequences says "Implementing this adds a model tier to catalog entries", so it's no longer stated as current. tier-conventions.md also gains a model tier row marked not yet implemented, and the Notes for review repeat that tier and vendor don't exist today.
| layer a key came from (the provenance gap ADR 0104 deferred); a lower layer | ||
| may only tighten `never_downgrade` and `allowed_models`. | ||
| - A rule can raise the review orchestrator's model; lowering it needs a later | ||
| ADR that amends #6527, on shadow `per_model_usage` evidence. Persona |
There was a problem hiding this comment.
MEDIUM — cites a closed issue as the amendable artifact. "a later ADR that amends #6527"
#6527 was closed on 2026-08-23 (the runtime-parity backlog). A closed issue cannot be amended, and a future ADR would have nothing to point at.
The rule it carried — the deciding session is never downgraded — is real and worth keeping; it is the citation that is stale. Note the same issue is cited live at :51 in Context, where "still holds" reads oddly for a closed backlog item.
Suggested edit: make this ADR the artifact — "…only by a later ADR that supersedes this clause" — and in Context cite #6527 as the origin of the rule rather than as something that "still holds".
There was a problem hiding this comment.
| fields at once. | ||
| 6. **Every choice is echoed** with the rules that set each field: plan block, | ||
| `metrics.json` (`routing.{mode,rules,applied,static_model,static_effort}`), | ||
| root-span attributes `fullsend.routing.*`, status-comment footer. In |
There was a problem hiding this comment.
MEDIUM — the echo records what applied, never what a guard suppressed, and §7's replay needs both.
:174-177 echoes "the rules that set each field" and §7 carries "choice, rules, task facts, cost, status, verdict". A rule that matched but whose effect never_downgrade, allowed_models, a persona floor or the served-model preflight rejected leaves no trace.
That is a real gap for this ADR's own learning loop: replay cannot distinguish "no rule matched" from "a rule matched and was suppressed", so a policy whose downgrades are all being silently rejected looks identical to one with no downgrades. Two of the example-config rules in the PR description are exactly that case, written by accident — which is the evidence that it happens.
(This is the one lesson worth importing from yetone/magpie, whose per-decision trace names which candidates were skipped and why.)
Suggested edit, one clause here:
…status-comment footer, and the rules that matched but whose effect a guard suppressed, naming the guard.
There was a problem hiding this comment.
Fixed in cd903f6. Decision item 4 now reads: the plan block, metrics.json, the root span and the status comment "name the rules that set each field and those a guard suppressed". The implementation is expected to have each suppressed rule name its guard in the echo and in the ledger, so replay can tell "no rule matched" apart from "a rule matched and was suppressed".
| **Per-prompt routing inside the sandbox** (the #6361 hooks). Rejected: the | ||
| sandbox sees only the prompt, which is author-controlled text; routing state | ||
| is per sandbox and lost; it is Claude-Code-only; and the decision comes after | ||
| the review has already read a diff it could have sized for free. |
There was a problem hiding this comment.
LOW — the mechanical argument against per-prompt routing is missing, and it is the strongest one.
The four grounds given are all governance arguments (author-controlled prompt, per-sandbox state, Claude-only, too-late). The decisive mechanical reason is absent: a mid-loop model change discards the accumulated prompt cache, and on vendors that seal reasoning content it is an outright error on the next tool result.
That is also the unstated reason behind :216's "lets a worker consult a stronger model without switching session model", so stating it here makes that later clause follow. It is the reason yetone/magpie holds one route across an entire turn and its whole tool-call loop, migrating only on context outgrowth.
Suggested edit, one clause:
…and a mid-loop model change discards the accumulated prompt cache — on some vendors an outright error on encrypted reasoning content — so the turn, not the prompt, is the smallest safe grain.
There was a problem hiding this comment.
Added in cd903f6. Options now rejects per-prompt routing because it "reads author-controlled prompts and loses the prompt cache mid-loop". To keep Options short in the trim, I dropped the encrypted-reasoning clause; I'm happy to put it back if you think it carries weight.
| `routing.trusted_labels` are empty, so no rule may name a model or key a | ||
| downgrade on a label. `fullsend run` validates `routing:` itself on every | ||
| run, as it does `agents:` (ADR 0091); a malformed block fails the run | ||
| before the sandbox exists rather than running without its escalations. |
There was a problem hiding this comment.
I think routing should be done by setting fields in overlays in the harness file and using the existing harness overlay "when" expressions, instead of separate rules in the config file.
If we want to allow repo-level overrides for the model selection, then it should be a part of a general repo-level overrides mechanism for harness settings, I think there is an in-flight ADR for such a mechanism already.
There was a problem hiding this comment.
Thanks. Rather than choose between the two designs here, I'm writing up the overlay approach as a second draft PR: routing as fields set in harness overlays with the existing when: expressions, under a general repo-level override mechanism for harness settings. I'm moving this PR to draft as well, so the team can compare the two options side by side and give feedback before either goes forward. This one keeps the separate routing: block so its guards (never-downgrade, allowlisted inputs, trust lists, the served-model preflight) can be judged on their own terms. Could you point me at the in-flight ADR for repo-level overrides you mentioned, so the overlay version builds on it?
…ecision Address the review on 3e2289e by @waynesun09, @ifireball and the review agent, then cut ADR 0112 from about 313 to 86 content lines so it records one decision, per skills/writing-adrs. Implementation detail is left to follow-up implementation work: the trusted-label and trusted-actor tests and the list of protected paths, the CEL environment and its edge cases, merge and preflight mechanics, echo escaping, runtime notes, and presets. Review follow-up: drop per-runtime inventories (codex personas, "fallback chain on Claude only", CLAUDE_CODE_SUBAGENT_MODEL and _FORCE); state the catalog model tier as a field to add; cite fullsend-ai#6527 only as the rule's origin; echo the rules a guard suppressed; and add the prompt-cache argument to Options. The overlays alternative is left to a separate rewrite, so the ADR no longer mentions ADR 0088. State at decision level what the trim must keep: - While any escalation matches, no field falls below static, and runtime or a swap within the same model tier keeps its static value. - An escalation keeps the runtime and changes a model only to a higher model tier; anything else is a downgrade. - never_downgrade agents keep their runtime and never lose model tier, while escalations may raise them and their personas. - Downgrades key only on evidence the change's author cannot supply and never apply to changes touching agent-instruction or policy files. - Rules override the agents: entry; flags and FULLSEND_* still win. - The served-list preflight runs only with routing on and fails a static model only when nothing in its fallback chain is served. - A downgrade rule reaches enforce only after replay evidence. Declare routing in repos.yaml for ADR 0122-managed repos, where any loosening is a relaxation under its safety gate. Align architecture.md and the problem docs with the trimmed text, shorten the decision-model open question, and add model tier to tier-conventions.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Tim Pouyer <tim.pouyer@gmail.com>
|
The overlay-based alternative @ifireball suggested is now up as draft #7842. Both PRs are drafts so we can compare the two approaches side by side. |
|
Code review by qodo was updated up to the latest commit cd903f6 |
|
Closing in favor of #7842 as discussed on Thursday Oct 1st in Technical Review Forum call. |
|
🤖 Finished Retro · ✅ Success · Started 2:27 PM UTC · Completed 2:44 PM UTC Commit: Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $2.90 |
|
Retro on #7808 ("docs(adr): route runtime and model per task from a config policy"), a human-authored draft ADR PR closed without merging on 2026-10-02 in favor of a competing design (#7842), per a live Technical Review Forum decision outside the automated pipeline. This is a normal, healthy design-competition outcome, not a pipeline failure. Review quality: the Separately, tracing the workflow runs surfaced a previously unfiled platform reliability bug: when the PR author submitted 10 GitHub reviews in rapid succession on 2026-09-29, all 10 resulting No evidence of rework caused by agent error, redundant agent token spend, or avoidable iteration — the multiple review rounds reflect genuine design iteration on a complex, competing-options ADR, not agent mistakes. Proposals filed |
Summary
Adds ADR 0112:
fullsend runresolves runtime, model, effort and persona models per task, on the host, before the sandbox exists, from arouting:block in.fullsend/config.yaml. Today every choice is static per agent (ADR 0091, ADR 0104), so a one-line docs change and a 2,000-line CI refactor get the same opus-at-high review. Docs only; no code.This PR supersedes #7251, as @waynesun09 suggested. His two commits are cherry-picked unchanged, keeping authorship and sign-off. Four follow-up commits address the reviews on #7251 and on this PR, and align the ADR with entity-first dispatch (ADR 0098) and declarative repo configuration (ADR 0122). The last one (everything since @waynesun09's review on
3e2289efc) also trims the ADR from about 313 to 86 content lines so it records one decision, perskills/writing-adrs. The implementation requirements it removed are left to the implementation work that follows once a design is chosen.Status: draft, one of two options. This PR is a draft alongside a second draft PR that decides the same question with the harness-overlay approach @ifireball suggested. Both are open so the team can compare the options and give feedback before either moves forward.
Related Issue
Refs:
fullsend-models.js: cross-provider fallback chain,phases, andbudget#6531: the routing advisor, deferredThis PR records the decision and closes none of these, since each still needs the implementation.
Changes
The decision, in brief
.fullsend/config.yaml. For repos that ADR 0122 manages, it's declared inrepos.yaml, and any change that loosens routing counts as a relaxation under 0122's safety gate.routing.rules[]withagents:, a CELwhen:, an optionalmode:and aset:of the four fields anagents:entry holds.routing.modeisoff(the default),shadoworenforce. A rule's own mode may act less than the block's, never more. A malformed block fails the run.FULLSEND_*variables ›routing.rules› the agent'sagents:entry › the repo default.routing.allowed_models(empty by default) and the catalog.never_downgrade(by defaulttriage,prioritizeandreview; the list can only grow) keep their runtime and keep or raise their model. Their personas keep their static model tier unless a persona file sets a lowermin_tier, and escalations may raise both.routing:andmin_tier:come from the base branch or pinned harness, never from the change being routed.metrics.json, the root span and the status comment name the rules that set each field and the rules a guard suppressed.routing:is the only path to production, and a downgrade rule moves toenforceonly after replay shows it never downgraded a PR whose review later requested changes.Options rejected:
An LLM gateway stays an optional provider that must report which model actually served.
Deferred: a routing advisor that reads the diff (#6531). An advisor escalation may fire on its own. A downgrade may use the advisor's assessment only as one condition, and the rule must still match with that condition removed. The advisor's classifier (such as a decision model like Jev or Laya) is an open question in
adaptive-agent-selection.md.Living documents updated
docs/architecture.md: one Decided line under Agent Runtime, marked not yet implemented.docs/problems/adaptive-agent-selection.mdandoperational-observability.md: annotations on the open questions this partly answers, plus the decision-model open question.docs/ADRs/0091-per-agent-runtime-model-effort.md: a forward link from its precedence chain.docs/contributing/tier-conventions.md: a new model tier row.Left to implementation
CLAUDE_CODE_SUBAGENT_MODELand the codex persona status)How it compares with #7842
routing:policy).fullsend/config.yamlrouting:block (orrepos.yamlunder ADR 0122)overlays:model,effortoff/shadow/enforce), trust listsnever_downgradelistReview follow-up
Round 1: review agent on #7251 (9 findings)
never_downgrademin_tierundefinedsubagentsreplaced as a whole fieldeventnot redactedmode: off, empty lists, a malformed block fails the runcross-run-memory.mdoutsiderelates_toRound 2: review agent on this PR (14 findings)
never_downgradepins only the orchestrator's modelbotservice actortask.actoris null on poll, schedule and manual runs, left to implementationtrusted_labelsandallowed_modelsarchitecture.mdstates the decision as shippedtier-conventions.mdRound 3: @waynesun09 and @ifireball
CLAUDE_CODE_SUBAGENT_MODELhas no effect on fullsend personas;_FORCE_FORCEdroppedconfidenceis not portableadaptive-agent-selection.mdLocal pr-review passes before pushing
The
pr-reviewpersona prompts (correctness, security, intent-coherence, docs-currency, style-conventions, risk-assessment, then the challenger) were run locally over twelve rounds, until only low findings remained.After the trim, those passes also led to these changes:
agents:entryThree low findings were declined:
architecture.mdbullet sits under (there's no decided-but-unimplemented list)Example config
An illustrative
.fullsend/config.yamlfor this repo under the ADR. The ADR fixes the concepts, not the schema: thetask.*field names, the catalog's tier names and the account ID below are placeholders.routing:block on this repo's configPersona floors live in the persona files, not in
config.yaml, and are read from the pinned harness. For an agent innever_downgrade, a persona with nomin_tieris floored at its static model tier:Config validation would reject these rules:
Testing
make lintpasses (stage changes first, then run). All pre-commit hooks pass on the six changed files, includinglint markdown links(lychee 0.24.2), the ADR status, number and frontmatter lints, and gitlint.Notes for review
when:expressions. This PR keeps the separaterouting:block and makes no claim about overlays either way, so the two can be compared side by side.tierandvendorto catalog entries, which don't exist today.main, and only this PR and docs(adr): route runtime and model per task from a config policy #7251 claim it. Re-check with therenumber-adrskill before merge.routing:schema with shadow mode; in-run escalation on pi and Claude Code; then replay and the cost preview.Checklist
!for breaking changes)🤖 Generated with Claude Code