Skip to content

Commit 9e31b29

Browse files
committed
refactor(evals): drop dispatch-routing narration
1 parent 9477471 commit 9e31b29

2 files changed

Lines changed: 2 additions & 92 deletions

File tree

Lines changed: 2 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -1,75 +1,14 @@
1-
# Dispatch routing eval (first `@corbits/evals` eval)
2-
3-
First eval wired through the published [`@corbits/evals`](https://github.com/corbitsdev/corbits-evals#readme)
4-
package (CL-10452). It exercises the **real dispatch director system prompt**
5-
(`agents/dispatch/src/prompt.ts`, exported by `@corbits/code-agent-dispatch`)
6-
against **brand-new routing requests**: each case is a user request that the
7-
dispatch director must route to one specialist.
8-
9-
## Run
1+
# Dispatch routing eval
102

113
```sh
124
bun scripts/eval-prompts-dispatch.ts
135
# or
146
bun run eval:prompts-dispatch
157
```
168

17-
The run calls a model through local Ollama by default (matching the package's
18-
own examples). Override with env:
9+
Local Ollama by default. Override with `EVAL_OLLAMA_URL` and `EVAL_MODEL`:
1910

2011
```sh
2112
EVAL_MODEL=tev1:0.8b bun scripts/eval-prompts-dispatch.ts
2213
EVAL_OLLAMA_URL=http://localhost:11434 EVAL_MODEL=qwen3:0.6b bun scripts/eval-prompts-dispatch.ts
2314
```
24-
25-
## What it measures
26-
27-
The run composes the dispatch card **verbatim** + the case request + a fixed
28-
reply-format stanza (harness glue so the routing decision is measurable), then
29-
asks the model which specialist it would spawn. Checks:
30-
31-
| Check | Helper | Passes when |
32-
| -------- | ---------- | -------------------------------------------------- |
33-
| `routes` | `mentions` | the reply names the expected specialist |
34-
| `noDiy` | `avoids` | the reply does not volunteer to do the work itself |
35-
| `fast` | `under` | the run finishes within 120s |
36-
37-
## Cases
38-
39-
`cases.jsonl` — 9 brand-new routing prompts. Each case:
40-
41-
```json
42-
{
43-
"name": "map-retry-backoff",
44-
"input": "Where does the repo handle retry backoff? Find every call site and sketch the flow between them.",
45-
"expect": { "route": "explorer", "decoys": ["coder", "planner"] }
46-
}
47-
```
48-
49-
`expect.route` is the correct specialist; `expect.decoys` lists the likely
50-
wrong picks for humans scanning the report.
51-
52-
## Gates
53-
54-
Per check, `allow` in `scripts/eval-prompts-dispatch.ts`:
55-
56-
- `routes: { miss: 6, drop: 0 }` — the gate trips only when fewer than 3 of 9
57-
cases route correctly. Calibrated to the local smoke model (`qwen3:0.6b`,
58-
observed 3/9); **tighten when running a capable model** (e.g. `miss: 1`).
59-
- `noDiy: { miss: 2 }`
60-
- `fast: 0`
61-
62-
`compare` + `save` point at `last.json`: each run compares drops against the
63-
previous stored run, then replaces it. Until a capable-model baseline is
64-
stored, drops are not gated.
65-
66-
## Integration notes
67-
68-
- `@corbits/evals@0.1.1` is a devDependency (LGPL-2.1-only — see the PR body).
69-
- The package's `evaluate()` is the whole harness: run, cases, checks, and
70-
per-check miss/drop allowances in one call. It rejects with `EvalFailure`
71-
when a check breaks its allowance; `.report.markdown()` prints the summary.
72-
- Jev (LLM-as-judge string/`ask` checks) needs the `@corbits/system-one`
73-
gateway; this eval uses code checks only so it runs offline.
74-
- Trajectory scoring (`ranSteps` / `@corbits/evals/workflow`) is the
75-
CL-10452 follow-up: this first eval uses a plain function run.

‎scripts/eval-prompts-dispatch.ts‎

Lines changed: 0 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,4 @@
11
#!/usr/bin/env bun
2-
/**
3-
* FIRST @corbits/evals eval: dispatch director routing (CL-10452).
4-
*
5-
* Exercises the real product dispatch director system prompt
6-
* (`agents/dispatch/src/prompt.ts`, exported by `@corbits/code-agent-dispatch`)
7-
* against brand-new routing requests. Each case is a user request that must be
8-
* routed to one specialist; the checks and the per-check miss allowances gate
9-
* the run through `evaluate()` from `@corbits/evals`.
10-
*
11-
* The run composes the dispatch card (unchanged) + the case request + a fixed
12-
* reply-format stanza (harness glue so the routing decision is measurable),
13-
* and calls a model. Defaults to local Ollama, matching the package's own
14-
* examples; override with EVAL_OLLAMA_URL / EVAL_MODEL.
15-
*
16-
* Run:
17-
* bun scripts/eval-prompts-dispatch.ts
18-
* EVAL_MODEL=tev1:0.8b bun scripts/eval-prompts-dispatch.ts
19-
*
20-
* See evals/prompts/dispatch-routing/README.md for gates and calibration.
21-
*/
222
import { readFile } from "node:fs/promises";
233
import { dirname, join } from "node:path";
244
import { fileURLToPath } from "node:url";
@@ -37,11 +17,9 @@ const LAST_PATH = join(REPO_ROOT, "evals/prompts/dispatch-routing/last.json");
3717
const OLLAMA_URL = process.env.EVAL_OLLAMA_URL ?? "http://localhost:11434";
3818
const MODEL = process.env.EVAL_MODEL ?? "qwen3:0.6b";
3919

40-
/** Harness glue appended after the case request; the dispatch card is verbatim. */
4120
const REPLY_FORMAT = `\n\n# Reply format
4221
Reply with exactly one line: the specialist you would spawn, one of: explorer, planner, coder, reviewer, designer, artist, warden, shakespeare, prober, qa-lead. Then a second line with a one-sentence brief.`;
4322

44-
/** Route one request: dispatch card + request (+ reply format) through the model. */
4523
async function dispatchRoute(
4624
input: string,
4725
signal: AbortSignal,
@@ -75,7 +53,6 @@ async function dispatchRoute(
7553
return reply;
7654
}
7755

78-
/** Per-case expected specialist and likely-wrong decoys (for humans; route gates). */
7956
interface RoutingExpect {
8057
route: string;
8158
decoys: readonly string[];
@@ -143,13 +120,11 @@ function loadCases(path: string): Promise<Case<string, RoutingExpect>[]> {
143120
});
144121
}
145122

146-
/** Fail-closed word list: never matches real output, so a missing expect fails. */
147123
const routeWords = (
148124
run: Run<string, string, RoutingExpect>,
149125
): readonly string[] =>
150126
run.expect === undefined ? ["__missing_expect__"] : [run.expect.route];
151127

152-
/** The dispatch card says delegate by default and never DIY-map explorer work. */
153128
const DIY_PHRASES = [
154129
"I will fix",
155130
"I'll fix",
@@ -178,10 +153,6 @@ async function main(): Promise<number> {
178153
noDiy: avoids<string, string, RoutingExpect>(DIY_PHRASES),
179154
fast: under({ ms: 120_000 }),
180155
},
181-
// Gates: calibrated to the local smoke model (qwen3:0.6b) so the harness
182-
// trips only on a routing collapse; tighten when running a capable model.
183-
// `last.json` (compare + save) is the regression/drop gate once a strong
184-
// model baseline is stored.
185156
allow: {
186157
routes: { miss: 6, drop: 0 },
187158
noDiy: { miss: 2, drop: 0 },

0 commit comments

Comments
 (0)