Pick the right image model for a creative brief in about 200ms,
with a confidence you can trust.
Home · Which model · Which image · Where in the frame
Media tools offer dozens of models, and users guess which one fits their brief. Typecast asks a decision model (TypeSafe's Jev, or a Jev-compatible open model) for a typed, calibrated choice. At 85% confidence or more the product just uses the pick; below that it shows the top two side by side. The same request asks a yes/no check per category, so a pick comes with its reason and an unsure route says what would settle it.
On 240 held-out briefs, never tuned against, 3 runs each (jev-1.14.0):
| 602 / 604 | 113 / 116 | 204ms | $0.032 |
|---|---|---|---|
| confident picks were the right model | unsure routes had the right model in the top two | median time to route a brief | per 1,000 routes |
Per set
| Held out | Accuracy | ECE | Log-loss | Confident misses | p50 | p95 |
|---|---|---|---|---|---|---|
test |
100% | 0.030 | 0.029 | 0 | 204ms | 273ms |
hard |
91.9% | 0.046 | 0.209 | 2 | 205ms | 319ms |
test was written blind to the briefs we iterate on.
hard hides its deciding cue, so a keyword baseline that scores 85-88% elsewhere gets 9% on it.
Against the original prompt, the current one lifts hard from 83.1% to about 92% and confident misses from 5.7 to under 1 per run (McNemar p = 0.003).
The two confident misses are one brief that sits on the 85% line, wrong in two of three runs.
The checks flipped no brief's majority choice on either set and added about 3ms; they are why a route now costs $0.032, not $0.026.
A route's own check agreed with it 96% of the time on test and 84% on hard; when it doesn't, the demo shows no reason at all.
Ten briefs, each rendered by all five local models with the same prompt and seed. Jev, blind to which model made which, picked the routed model's image 7 times, the default's twice (chance is 2). A second look is less kind: scoring each image alone on a rubric (brief, style, craft, and text when asked for) gave the routed model the highest overall score 4 times and the default 3, and the top-scoring image was the pick on 7 of 10. A small sample of briefs routing was tuned on, judged by Jev rather than people: a signal, not proof.
Name a target and one Jev request returns its box and whether it is there at all. The page cuts a padded square around it, ready for a face or detail pass, in about 0.35s for $0.00016. Box accuracy isn't measured yet; on nine samples it boxed every visible face.
cp .env.example .env # add your TYPESAFE_API_KEY
npm install
npm run demo # local web UI at http://localhost:4180
npm run route -- "Hero image for a spring sneaker campaign, studio lighting"Point TYPESAFE_BASE_URL at a local Jev-compatible server (Kev, Laya, ...) to run without the hosted API.
Evaluate a change
npm run eval # accuracy, calibration, latency, cost
npm run eval -- --data data/test.jsonl --repeats 3
npm run compare -- runs/<default>.json runs/<candidate>.json # paired deltas, bootstrap CIs, McNemarECE (Expected Calibration Error) asks whether 90% confidence is right 90% of the time.
dev is for iterating; test and hard are held out and scored only on final candidates.
Render and judge images (Apple Silicon)
uv tool install mflux
npm run generate # routed vs default model on 50 dev briefs
npm run generate -- --shootout # every model on every brief
npm run judge -- runs/level2/<timestamp> # blind side-by-side judging by hand
npm run judge-jev -- runs/level2/<timestamp> # Jev judges every pair (about $0.0005 a pair)
npm run judge-report -- runs/level2/<timestamp> # routed win rate with a 95% CI
npm run pick-jev -- runs/shootout/<timestamp> # Jev picks the best of each briefSet HF_XET_HIGH_PERFORMANCE=1 for the first generate: it downloads about 100GB of weights.
Host the site
npm run site # writes site/ and bakes the samples' answers into build/
vercel deploy # a preview; add --prod to publishHosted, there is no Mac to render on, so the site routes and finds but doesn't render.