diff --git a/docs/README.md b/docs/README.md
index b6985af..ebbb345 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -9,9 +9,9 @@ folder `/docs`.
| Path | What it is |
| --- | --- |
-| `index.html` | Landing page: why the experiments were run, what System One models are, the four experiments, the protocol, the notebook log, the limits |
+| `index.html` | Landing page: why the experiments were run, the premise (what Jev is), the four experiments, the process, the notebook log, a note on the level of rigor |
| `assets/` | Shared by every page: styles (`site.css`) and the theme toggle (`site.js`). Shared by the results pages: `results.css`, and `results.js` (tooltips, chart helpers, the side contents) |
-| `model-router/` | The 2026-10-08 model-router run: `index.html`, `app.js`, and generated `data.js` |
+| `model-router/` | The 2026-10-08 model-router run: `index.html`, `app.js`, generated `data.js`, and two captures of the page's route explorer (`screenshot-light.png`, `screenshot-dark.png`) that the landing page shows as a thumbnail |
| `godot-jev/` | The 2026-10-08 three-model godot-jev run: `index.html`, `page.css`, `app.js`, a screenshot of the game, and generated `data.js` |
| `tools/build-model-router-data.mjs` | Builds `model-router/data.js` |
| `tools/build-godot-jev-data.gd` | Builds `godot-jev/data.js` |
diff --git a/docs/godot-jev/index.html b/docs/godot-jev/index.html
index fb19090..9be6c74 100644
--- a/docs/godot-jev/index.html
+++ b/docs/godot-jev/index.html
@@ -73,8 +73,9 @@
Three Jevs parse 16 lines of player slang.
model, with the moves possible right now as the options of one Choice. I gave three Jevs, one hosted
and two running on a laptop, the same 16 lines in the same game states, three times each. All three got
the player to the crown without a wrong move. They differ in how sure they were, how often the game had
- to ask what the player meant, and how long a turn took on this setup. This is a gut check on a small,
- hand-written set of lines, not a benchmark: what it does and doesn't show .
+ to ask what the player meant, and how long a turn took on this setup. This is a gut check on a small
+ set of lines written for a recording, not a benchmark:
+ what it does and doesn't show .
diff --git a/docs/index.html b/docs/index.html
index 82c3d18..a896659 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -351,6 +351,51 @@
font-size: 0.78rem;
color: var(--muted);
}
+ .with-thumb {
+ display: grid;
+ grid-template-columns: 200px minmax(0, 1fr);
+ gap: 18px;
+ align-items: start;
+ margin-top: 16px;
+ }
+ .thumb {
+ margin: 0;
+ }
+ .thumb a {
+ display: block;
+ border: 1px solid var(--rule);
+ border-radius: 8px;
+ overflow: hidden;
+ box-shadow: var(--shadow);
+ }
+ .thumb a:hover {
+ border-color: var(--rule-strong);
+ }
+ .thumb img {
+ display: block;
+ width: 100%;
+ height: auto;
+ }
+ .thumb figcaption {
+ margin-top: 6px;
+ }
+ .thumb .only-dark {
+ display: none;
+ }
+ @media (prefers-color-scheme: dark) {
+ :root:not([data-theme="light"]) .thumb .only-light {
+ display: none;
+ }
+ :root:not([data-theme="light"]) .thumb .only-dark {
+ display: block;
+ }
+ }
+ :root[data-theme="dark"] .thumb .only-light {
+ display: none;
+ }
+ :root[data-theme="dark"] .thumb .only-dark {
+ display: block;
+ }
@media (max-width: 860px) {
.specimens,
.specimen.featured .grid {
@@ -358,7 +403,8 @@
}
}
@media (max-width: 480px) {
- .finding-row {
+ .finding-row,
+ .with-thumb {
grid-template-columns: 1fr;
}
}
@@ -494,6 +540,21 @@
margin-top: 0.9em;
}
+ .ai-note {
+ max-width: var(--measure);
+ margin-top: 24px;
+ padding: 14px 16px;
+ border: 1px solid var(--rule);
+ border-radius: var(--radius);
+ background: var(--surface);
+ box-shadow: var(--shadow);
+ }
+ .ai-note p {
+ margin: 4px 0 0;
+ font-size: 0.92rem;
+ color: var(--ink-2);
+ }
+
.limits {
display: grid;
grid-template-columns: minmax(0, 1fr) minmax(0, 1fr);
@@ -528,7 +589,7 @@
Experiments
- Protocol
+ Process
Notebook
Latest results
GitHub
@@ -588,22 +649,25 @@ What can a model do when it can't write?
Why I ran these
- I wanted a quick, reasonably informed gut check on whether this class of model is viable. The jobs I
- had in mind are small decisions that fixed rules can't make: which move a player meant, which model a
- sub-agent task needs, which misconception an answer shows.
-
-
- Each experiment follows the same method. It states a hypothesis and the checks that would confirm it,
- pins the model version, records every answer, and builds its results page from those records. The
- rigor still stops well short of a study. The samples are small and hand-written, the labels are one
- person's opinion, and most of the checks are open. The limits are listed below.
+ I wanted a quick, reasonably informed gut check on whether this class of model is viable for small
+ decisions that fixed rules can't make: which move a player meant, which model a sub-agent task needs,
+ which misconception an answer shows.
- So far two of the four experiments have live runs. In both, a hosted model and two running on a laptop
+ Two of the four experiments have live runs so far. In both, a hosted model and two running on a laptop
handled a small set of mostly clear cases without a costly mistake, and differed in how sure they
- were. That tells me the mechanism works. It doesn't give an accuracy figure. If one of these is close
- to a problem you have, the code is there to run against your own data.
+ were. That tells me the mechanism works. It isn't an accuracy figure, and the
+ note on rigor says why.
+
+ On the use of AI
+
+ I leverage AI (with very close attention from myself), for the code and the documentation, this page
+ included. I get real value from the write-ups: they are how I come to understand what a run showed.
+ It is never my intention to publish slop, so if something here is wrong or reads as filler, please
+ open an issue .
+
+
@@ -614,12 +678,17 @@ Why I ran these
02
-
Three kinds of question, no prose
+
The premise: what Jev is
- Jev is TypeSafe's System
- One model, and Open Jev serves the same
- /v1/systemone API from open weights on your own machine. You send state and questions; you
- get back distributions. The code around it decides what to do with them.
+ Jev is TypeSafe's System One model. It never writes a reply. You send it some state and typed
+ questions about that state, it returns a probability for every possible answer, and ordinary code
+ decides what to do with those numbers. Open Jev serves
+ the same API from open weights on your own machine.
+
+
+ It takes three kinds of question, and the experiments below use all three.
+ TypeSafe's introduction
+ covers the rest.
@@ -661,7 +730,8 @@ The experiments
Each is a standalone directory with its own stack, a hypothesis, the checks that would confirm it, and
tests that run without an API key. Skeleton means the design is in code and tested against
- scripted answers. Live means runs against real models are recorded, with dates.
+ scripted answers. Live means runs against real models are recorded, with dates. If one is
+ close to a problem you have, the code is there to run against your own data.
@@ -679,10 +749,33 @@ Which model should this sub-agent run on?
An agent harness asks one Score question about a sub-agent's task, then takes the cheapest model
whose chance of being too weak is acceptable. The orchestrator spends no tokens deciding.
-
- One Jev call over the text of a sub-agent task chooses a model tier well enough to move a large
- share of work off the most capable model, while rarely sending a task to a model too weak for it.
-
+
+
+
+
+
+
+ One decision, from the results page
+
+
+ One Jev call over the text of a sub-agent task chooses a model tier well enough to move a large
+ share of work off the most capable model, while rarely sending a task to a model too weak for it.
+
+
First live run · 2026-10-08 · 24 tasks
@@ -700,7 +793,7 @@
Which model should this sub-agent run on?
median decision, TypeSafe Jev hosted (on one laptop: Flash 9B 181, 27B 567)
- Timings describe this setup, not how fast the models are .
+ Timings describe this setup, not how fast the models are .
Walk through the results →
Code
@@ -721,10 +814,24 @@
The Sunken Crypt
A dungeon crawl whose parser understands whatever you type but can only choose moves the author
wrote. Each turn, the possible moves become one Choice, plus "none of these".
-
- Players type naturally with few dead ends, the game stays fully authored, and it's fast enough to
- feel like a parser, not a chat.
-
+
+
+
+
+
+ The game, asking "did you mean?"
+
+
+ Players type naturally with few dead ends, the game stays fully authored, and it's fast enough to
+ feel like a parser, not a chat.
+
+
Three-model run · 2026-10-08 · 16 lines × 3 rounds
@@ -742,7 +849,7 @@
The Sunken Crypt
median turn, TypeSafe Jev hosted (on one laptop: Flash 9B 185, 27B 579)
- Timings describe this setup, not how fast the models are .
+ Timings describe this setup, not how fast the models are .
-
+
04
-
Protocol
+
More about my process
+
+ Each experiment's README states a hypothesis, the checks that would confirm it, and what is still
+ open. An AI coding agent does most of the building, tests first, working to a short set of rules kept
+ in the repository's
+ AGENTS.md .
+ One of them: loosening a check, or changing what an experiment is trying to show, is my decision and
+ not the agent's.
+
- Every experiment is built the same way, so the model can be swapped and no test needs the network.
+ A live run is a script that pins the model version and writes every answer to a file. Each results
+ page is generated from those files, and the generator refuses to write if its totals differ from the
+ ones the run saved. When a check gets evidence, the README records it with the date.
+
+
+ The code follows the same five rules in every experiment, so the model can be swapped and no test
+ needs the network.
A decider port
- Code asks decide(state, questions). One adapter calls Jev; a scripted one drives the tests. No unit test calls the network.
+ Code asks decide(state, questions) and gets typed answers back. One adapter calls Jev and a scripted one drives the tests, so no unit test calls the network or needs a key.
Policy in code
- Thresholds, arithmetic, dates and counting stay in ordinary code. A policy change is a code change, not a prompt rewrite.
+ Thresholds, arithmetic, dates and counting stay in ordinary code. The model only makes the judgments, so a policy change is a code change, not a prompt rewrite.
Ask only what's open
- Deterministic rules go first. The model answers only what they leave undecided.
+ Deterministic rules go first: a Conventional Commit prefix, the moves possible in a room. The model answers only what they leave undecided.
Confidence decides the path
- Act, ask or escalate by how sure the model is, with the bar set by the cost of being wrong.
+ Act, ask or escalate by how sure the model is, with the bar set by the cost of being wrong. The crypt asks "did you mean?"; the router moves a task up a tier.
Fail safe
- If the model is unreachable, fall back to what the system did before it had one.
+ If the model is unreachable, each experiment falls back to what it did before: an authored "not possible" line, Conventional Commits only, the harness's default model.
@@ -899,32 +1020,37 @@ Notebook
-
+
06
-
Read before trusting
-
Limits of the evidence, and of the models, that apply to every experiment here.
+
A note on the current level of rigor
+
+ The goal was a gut check, so the rigor is pragmatic: enough to tell whether an idea deserves more
+ work, not enough to quote a number from. I plan to formalize it as the experiments continue, and
+ each README's Open list says what comes next.
+
- The evidence is thin. The two live runs cover 24 tasks and 16 lines, hand-written for these
- experiments, and most are clear cases. Nobody else has checked the labels, and nothing has been run on
- input from real users.
+ The samples are small. The live runs cover 24 tasks and 16 lines, written for these experiments
+ and mostly clear cases.
+
+
+ AI did most of the work. It wrote the code and these pages, and in model-router it also wrote
+ and labelled the tasks. Nobody else has checked the labels.
+
+
+ The timings describe one laptop , not how fast the models are. Each results page gives the
+ setup.
- Jev takes text only and reads instructions literally.
- It can't count or compare dates. That's why arithmetic and thresholds live in code.
- Text inside the state can argue for an answer: a learner, a commit author, a player or an orchestrating model can write "this is trivial".
- Where the state comes from someone with a reason to game it, keep the stakes of any automatic action low, or keep a human in the loop.
- The latencies for the local models are not a statement of how fast those models are. They come
- from 4-bit MLX builds on one Apple M5 Pro laptop with other applications open. Server hardware, another
- quantisation or another runtime would give different numbers. The hosted figures include a network
- round trip from that laptop. Read every timing here as what a decision cost in this setup.
+ Jev has limits of its own. It can't count or compare dates, and text in its input can argue for
+ an answer, so keep the stakes of any automatic action low.
diff --git a/docs/model-router/app.js b/docs/model-router/app.js
index 4b20143..a9485b3 100644
--- a/docs/model-router/app.js
+++ b/docs/model-router/app.js
@@ -50,7 +50,7 @@
const offTop = runs.map((r) => N - atDefault(r).routes[ids[TOP]]);
const sameOff = offTop.every((v) => v === offTop[0]);
$("#hero-stats").innerHTML = `
- ${N} tasks
hand-written sub-agent tasks, each asked once of ${runs.length} decision models
+ ${N} tasks
sub-agent tasks written and labelled by Claude, each asked once of ${runs.length} decision models
${under}
tasks sent to a model too weak for them, by any of the three
${sameOff ? pct(offTop[0]) : offTop.map(pct).join(" / ")}
of tasks kept off the most capable model${sameOff ? ", by all three" : ""}. ${misses} miss in ${N * runs.length} decisions.
${runs
@@ -73,7 +73,7 @@
"1 · Routing",
"On ≥ 100 real tasks, a bar where ≤ 5% go too low and ≥ 50% leave the top route",
s.map((x) => `
${pct(x.under)} low · ${pct(N - x.routes[ids[TOP]])} off top `),
- `
Not yet evidence 24 starter tasks, one author `,
+ `
Not yet evidence 24 starter tasks, all written by Claude `,
],
[
"2 · Speed",
diff --git a/docs/model-router/index.html b/docs/model-router/index.html
index c3efdee..e9cb476 100644
--- a/docs/model-router/index.html
+++ b/docs/model-router/index.html
@@ -71,7 +71,7 @@
Three Jevs pick the model for 24 sub-agent tasks.
One request to a System One model decides whether a sub-agent runs on Haiku, Sonnet or Opus. I asked
three Jevs, one hosted and two running on a laptop, to make that call for the same tasks. Their routes
nearly match. They differ in how sure they were, and in how long an answer took on this setup. This is
- a gut check on a small, hand-written set of tasks, not a benchmark:
+ a gut check on a small set of tasks that Claude wrote and labelled, not a benchmark:
what it does and doesn't show .
@@ -462,7 +462,7 @@
What this does and doesn't show
It doesn't show
24 tasks, asked once each. One task is 4 percentage points.
- One author wrote everything. The tasks, their labels and the route descriptions came from one sitting, so tasks echo the wording they're judged against ("find the cause" against "a failure with an unknown cause").
+ One author wrote everything. Claude wrote the tasks, their labels and the route descriptions in one sitting, so tasks echo the wording they're judged against ("find the cause" against "a failure with an unknown cause").
Labels are opinions. Nobody ran these tasks on the three Claude models to see which was enough. That needs a harness (check 4).
Flash 9B's 24 of 24 has little margin. On tasks 13 to 16 it was close to an even split between two routes.
The tasks don't separate the models. All three are at or near 24 of 24, so this says nothing about which decides best.
diff --git a/docs/model-router/screenshot-dark.png b/docs/model-router/screenshot-dark.png
new file mode 100644
index 0000000..1f37ba8
Binary files /dev/null and b/docs/model-router/screenshot-dark.png differ
diff --git a/docs/model-router/screenshot-light.png b/docs/model-router/screenshot-light.png
new file mode 100644
index 0000000..39efaa2
Binary files /dev/null and b/docs/model-router/screenshot-light.png differ
diff --git a/model-router/eval/results-2026-10-08.md b/model-router/eval/results-2026-10-08.md
index f616c95..18ce560 100644
--- a/model-router/eval/results-2026-10-08.md
+++ b/model-router/eval/results-2026-10-08.md
@@ -6,9 +6,10 @@ sources: `results-2026-10-08T16-41-27-867Z.json` (27B), `results-2026-10-08T18-4
An agent harness has to pick a model each time its orchestrator spawns a sub-agent. model-router
makes that choice with one request to a System One model (Jev), which returns probabilities, not
-text. This is the first live run: 24 hand-written tasks, each sent once to three decision models.
-One is TypeSafe's hosted Jev. The other two are [Open Jev](https://huggingface.co/openjev) models
-running locally: the 27B model, and Flash 9B, a smaller one released the day before.
+text. This is the first live run: 24 tasks that Claude wrote and labelled, each sent once to three
+decision models. One is TypeSafe's hosted Jev. The other two are
+[Open Jev](https://huggingface.co/openjev) models running locally: the 27B model, and Flash 9B, a
+smaller one released the day before.
## Result