Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ confirm it, and tests that run without an API key. The first three were picked f
| Experiment | The idea | Stack | Status |
| --- | --- | --- | --- |
| [misconception-gates](misconception-gates/) | Grade a learner's free-text answer by *which misconception* it shows, and jump to the lesson section that fixes it | Node, no dependencies (drops into js-animation-sandbox) | Skeleton |
| [godot-jev](godot-jev/) | A Godot 4 add-on for Jev, and a text adventure whose parser understands anything but can only choose moves the author wrote | Godot 4.7, GDScript | Live: TypeSafe Jev and Open Jev |
| [godot-jev](godot-jev/) | A Godot 4 add-on for Jev, and a text adventure whose parser understands anything but can only choose moves the author wrote | Godot 4.7, GDScript | Live: TypeSafe Jev, Open Jev 27B and Flash 9B, on 16 scripted lines |
| [semver-judge](semver-judge/) | Decide the next version from the commits since the last tag, flagging only the uncertain commits that could change the answer | Python, GitHub Action | Skeleton |
| [model-router](model-router/) | Pick the model for a sub-agent task in an agent harness: rules first, then a Score over the configured routes, moving up when a weaker model is too likely to fall short | Node, no dependencies | Live: TypeSafe Jev, Open Jev 27B and Flash 9B, on starter tasks only |

Expand Down Expand Up @@ -53,8 +53,12 @@ cd model-router && node --test && node eval/run.mjs
```

godot-jev and model-router can use a local Open Jev server instead (`JEV_BACKEND=openjev`).
godot-jev's `eval/compare_backends.gd` asks both the same questions, and model-router's
`eval/compare.mjs` puts runs side by side; see their READMEs.
godot-jev's `eval/compare_backends.gd` asks every backend the same questions, and model-router's
`eval/compare.mjs` puts runs side by side; see their READMEs. Both recorded runs are walked
through on the [docs site](https://alteredcraft.github.io/system-one-experiments/).

Latencies recorded for the local models describe one laptop (4-bit MLX builds on an Apple M5 Pro,
macOS, other applications open), not how fast those models are.

## Limits that apply to all of them

Expand Down
17 changes: 15 additions & 2 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,11 @@ folder `/docs`.
| Path | What it is |
| --- | --- |
| `index.html` | Landing page: what System One models are, the four experiments, the protocol, the notebook log |
| `assets/` | Shared styles (`site.css`) and the theme toggle (`site.js`) |
| `model-router/` | The 2026-10-08 model-router run: `index.html`, `results.css`, `app.js`, and generated `data.js` |
| `assets/` | Shared by every page: styles (`site.css`) and the theme toggle (`site.js`). Shared by the results pages: `results.css`, and `results.js` (tooltips, chart helpers, the side contents) |
| `model-router/` | The 2026-10-08 model-router run: `index.html`, `app.js`, and generated `data.js` |
| `godot-jev/` | The 2026-10-08 three-model godot-jev run: `index.html`, `page.css`, `app.js`, a screenshot of the game, and generated `data.js` |
| `tools/build-model-router-data.mjs` | Builds `model-router/data.js` |
| `tools/build-godot-jev-data.gd` | Builds `godot-jev/data.js` |

`model-router/data.js` is generated, not written by hand. The script reads the three recorded runs in
`model-router/eval/results-2026-10-08T*.json`, routes every answer with model-router's own policy
Expand All @@ -23,4 +25,15 @@ ones each run saved. After a change to those runs or to the policy, regenerate i
node docs/tools/build-model-router-data.mjs
```

`godot-jev/data.js` is generated the same way, by a Godot script, because the policy it replays is
the game's parser. It reads `godot-jev/eval/results-2026-10-08T134205.json`, puts every recorded
answer back through `demo/parser.gd` in the game state where it was asked, at the run's thresholds
and at every setting the page's sliders reach, and refuses to write if an outcome or a summary
differs from what the run saved:

```bash
godot --headless --path godot-jev --import
godot --headless --path godot-jev --script ../docs/tools/build-godot-jev-data.gd
```

To preview locally: `python3 -m http.server --directory docs`.
33 changes: 32 additions & 1 deletion docs/model-router/results.css → docs/assets/results.css
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
/* model-router results page. Shared tokens are in ../assets/site.css. */
/* Shared by the results pages in docs/. Tokens are in site.css. */

@media (max-width: 520px) {
.hide-xs {
Expand Down Expand Up @@ -1210,6 +1210,37 @@ input[type="range"] {
}
}

/* A note the reader needs before the figures beside it. */
.callout {
background: color-mix(in srgb, var(--warning) 8%, var(--surface));
border: 1px solid color-mix(in srgb, var(--warning) 45%, var(--rule));
border-left-width: 3px;
border-radius: var(--radius);
padding: 16px 20px;
margin: 0 0 20px;
max-width: 900px;
font-size: 0.95rem;
color: var(--ink-2);
}
.callout .label {
display: block;
margin-bottom: 6px;
color: var(--warning-text);
}
.callout p {
margin: 0 0 0.6em;
}
.callout p:last-child {
margin-bottom: 0;
}
.callout b {
color: var(--ink);
font-weight: 600;
}
.hero-stats .k a {
white-space: nowrap;
}

/* Caveats and details */
.caveats {
display: grid;
Expand Down
158 changes: 158 additions & 0 deletions docs/assets/results.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
/* Shared by the results pages: tooltips, the scoreboard, the latency lanes,
chart scales, the side contents with the reading-progress bar, and
redrawing charts on resize.
Load it after the page's #tip element and before the page's own script. */

window.S1X = (function () {
"use strict";

const $ = (sel) => document.querySelector(sel);
const esc = (s) => String(s).replace(/[&<>"]/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;" })[c]);
const f2 = (x) => x.toFixed(2);
const mColor = (run) => `var(--m-${run.id})`;
const swatch = (run) => `<span class="sw" style="--c:${mColor(run)}"></span>`;

/* ---------- tooltip: any element with data-tip ---------- */
const tip = $("#tip");
const tipAttr = (html) => `data-tip="${esc(html)}" tabindex="0"`;
function placeTip(x, y) {
const pad = 14;
const w = tip.offsetWidth;
const h = tip.offsetHeight;
let left = x + pad;
let top = y + pad;
if (left + w > innerWidth - 8) left = Math.max(8, x - w - pad);
if (top + h > innerHeight - 8) top = Math.max(8, y - h - pad);
tip.style.left = `${left}px`;
tip.style.top = `${top}px`;
}
function showTip(el, x, y) {
tip.innerHTML = el.dataset.tip;
tip.hidden = false;
if (x == null) {
const r = el.getBoundingClientRect();
x = r.left + r.width / 2;
y = r.bottom;
}
placeTip(x, y);
}
document.addEventListener("pointerover", (e) => {
const el = e.target.closest("[data-tip]");
if (el) showTip(el, e.clientX, e.clientY);
});
document.addEventListener("pointermove", (e) => {
if (!tip.hidden && e.target.closest("[data-tip]")) placeTip(e.clientX, e.clientY);
});
document.addEventListener("pointerout", (e) => {
if (e.target.closest("[data-tip]") && !e.relatedTarget?.closest?.("[data-tip]")) tip.hidden = true;
});
document.addEventListener("focusin", (e) => {
const el = e.target.closest("[data-tip]");
if (el) showTip(el);
});
document.addEventListener("focusout", () => (tip.hidden = true));
addEventListener("scroll", () => (tip.hidden = true), { passive: true });

/* ---------- small SVG helpers ---------- */
const lin = (d0, d1, r0, r1) => (v) => r0 + ((v - d0) / (d1 - d0)) * (r1 - r0);
const widthOf = (el) => Math.max(280, el.clientWidth);

/* One card per run, each metric beside its difference from `ref`. A metric is
{ k, v(run), f(value), better: 1 | -1 | 0 } with optional key, frac
(a fraction, shown to two places) and ratio (shown as a multiple of ref). */
function scoreCards(runs, ref, metrics) {
return runs
.map((r) => {
const rows = metrics
.map((m) => {
const v = m.v(r);
let d = "";
let worse = false;
if (r !== ref) {
const rv = m.v(ref);
if (m.ratio) {
d = `×${(v / rv).toFixed(1)}`;
worse = v > rv;
} else {
const diff = v - rv;
const near = Math.abs(diff) < (m.frac ? 0.005 : 0.5);
d = near ? "=" : `${diff > 0 ? "+" : "−"}${m.frac ? f2(Math.abs(diff)) : Math.abs(diff)}`;
worse = !near && m.better !== 0 && Math.sign(diff) === -m.better;
}
}
return `<div class="r${m.key ? " key" : ""}"><span class="k">${m.k}</span><span class="v">${m.f(v)}</span><span class="d${worse ? " worse" : ""}">${d}</span></div>`;
})
.join("");
return `<article class="score" style="--c:${mColor(r)}"><h3>${esc(r.name)}${r === ref ? '<span class="ref">reference</span>' : ""}</h3><div class="rows">${rows}</div></article>`;
})
.join("");
}

/* One lane per run with a dot per row, ticks at the run's latency.p50 and
latency.p95, and the speed check as a dashed line. `time(row)` is a row's
milliseconds, `tip(run, row, i)` its tooltip, `per` what was timed. */
function laneChart(el, runs, { time, tip, per, checkMs }) {
const W = widthOf(el);
const narrow = W < 560;
const rowH = 84;
const m = { l: narrow ? 70 : 130, r: 16, t: 26, b: 36 };
const H = m.t + rowH * runs.length + m.b;
const maxMs = Math.ceil(Math.max(...runs.map((r) => r.latency.max), checkMs) / 100) * 100;
const x = lin(0, maxMs, m.l, W - m.r);
let g = "";
for (let t = 0; t <= maxMs; t += 100) {
g += `<line class="grid" x1="${x(t)}" x2="${x(t)}" y1="${m.t}" y2="${H - m.b}"/><text class="tick-text" x="${x(t)}" y="${H - m.b + 16}" text-anchor="middle">${t}</text>`;
}
g += `<text x="${W - m.r}" y="${H - 4}" text-anchor="end">ms per ${per} →</text>`;
g += `<line class="check-line" x1="${x(checkMs)}" x2="${x(checkMs)}" y1="${m.t - 10}" y2="${H - m.b}"/><text class="ann" x="${x(checkMs) - 6}" y="${m.t - 12}" text-anchor="end">check 2: p95 ≤ ${checkMs} ms</text>`;
let body = "";
runs.forEach((run, k) => {
const cy = m.t + rowH * k + rowH / 2;
body += `<text class="ann-strong" x="${m.l - 12}" y="${cy + 4}" text-anchor="end">${esc(narrow ? run.short : run.name)}</text>`;
body += `<line class="axis" x1="${m.l}" x2="${W - m.r}" y1="${cy + rowH / 2}" y2="${cy + rowH / 2}" style="stroke:var(--rule)"/>`;
run.rows.forEach((row, i) => {
const jitter = ((i * 7) % 5) - 2;
body += `<circle class="dot" cx="${x(time(row))}" cy="${cy + jitter * 4}" r="4.5" style="fill:${mColor(run)};fill-opacity:.8" ${tipAttr(tip(run, row, i))}/>`;
});
// p50 labelled above its tick, p95 below, so close ticks don't collide.
for (const [k2, v, ty] of [
["p50", run.latency.p50, cy - 24],
["p95", run.latency.p95, cy + 32],
]) {
body += `<line x1="${x(v)}" x2="${x(v)}" y1="${cy - 20}" y2="${cy + 20}" style="stroke:var(--ink);pointer-events:none" stroke-width="2"/><text class="ann" x="${x(v)}" y="${ty}" text-anchor="middle" style="font-size:10px">${k2}</text>`;
}
});
el.innerHTML = `<svg viewBox="0 0 ${W} ${H}" role="img" aria-label="Time per ${per} for each model">${g}${body}</svg>`;
}

function tocAndProgress() {
const links = [...document.querySelectorAll("#toc a")];
const secs = links.map((a) => document.querySelector(a.getAttribute("href")));
const bar = $("#progress");
const onScroll = () => {
const h = document.documentElement;
bar.style.width = `${(h.scrollTop / Math.max(1, h.scrollHeight - h.clientHeight)) * 100}%`;
let active = -1;
secs.forEach((s, i) => {
if (s.getBoundingClientRect().top < innerHeight * 0.35) active = i;
});
links.forEach((a, i) => a.classList.toggle("active", i === active));
};
addEventListener("scroll", onScroll, { passive: true });
onScroll();
}

/* Charts are sized to their container, so they are redrawn when the width changes. */
function redrawOnResize(draw) {
let timer;
let lastW = innerWidth;
addEventListener("resize", () => {
if (innerWidth === lastW) return;
lastW = innerWidth;
clearTimeout(timer);
timer = setTimeout(draw, 150);
});
}

return { $, esc, f2, mColor, swatch, tipAttr, lin, widthOf, scoreCards, laneChart, tocAndProgress, redrawOnResize };
})();
Loading
Loading