Skip to content

feat(modelchk): name the task domain a tier is measured in - #24

Merged
LilMGenius merged 6 commits into
feat/falsifiable-reflexesfrom
feat/modelchk-task-domain
Nov 2, 2026
Merged

LilMGenius merged 6 commits into
feat/falsifiable-reflexesfrom
feat/modelchk-task-domain

Conversation

@LilMGenius

@LilMGenius LilMGenius commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

modelchk sized a tier and an effort but never said what a tier is measured against, so a binder mapping tiers to models had only price to go on. When two vendors price their lineups the same, price rank picks whichever model is listed first, and every role drifts to one vendor. Role auto-assign in opencodex did exactly that until lidge-jun/opencodex#7624 tiered candidates by measured capability.

Failing input. Sizing any task with modelchk returns recommended_tier and recommended_effort and nothing that tells a binder which measured strength the tier depends on. A binder given two equally priced candidates, one strong at coding and one at writing, has no basis in the skill to prefer either for a coding task.

Cause. skills/depth/modelchk/SKILL.md defined a tier only as "the cheapest sufficient class" and stopped at the two dials. It said nothing about the capability axis a tier is measured on or how a binder turns a tier into a model.

Change.

  • A third output, task_domain (coding, agentic, reasoning, writing, knowledge, general). It sizes nothing; it names the measured strength the tier depends on. A workflow step picks it, the report carries it, and the verification checklist requires exactly one.
  • A binder paragraph: a tier resolves against measured capability in that domain, read from an evaluation source the operator has enabled, with price choosing only among models that measure sufficient. Equal measurements give an equal tier whatever the listing order, and one model behind two routes counts once. At equal price the higher measurement wins before listing order, and a measured model with no known price stays eligible after the priced ones. Price rank stands in only when no source is enabled, the source cannot be read, or it measures fewer than two candidates, and the binder says which.
  • Three limits on any source: an explicit user mapping outranks measurement and price rank; a model the source does not list is unmeasured, not weak, and the binder says so; the source's terms decide whether scores may be shown or only used internally.
  • The no-vendor rule now also bars naming evaluation sources, and the README fix entry mentions the domain.

The tie, fallback, opt-in and unpriced sentences come from defects reviewers found in #7624 before it merged: equal scores split across tiers by provider order, a fallback described more narrowly than the code ran it, an unpriced measured model documented as never proposed, and a benchmark download the owner ruled must be opt-in.

Verification (on this branch at the head of this PR):

Command Result
node scripts/gates/llms.cjs exit 0, llms-full.txt regenerated
npm run check exit 0: skills, catalog, links, prose, llms ok
npm test 235 pass, 0 fail, 1 skipped
uv run --with pyyaml python <skill-creator>/scripts/quick_validate.py skills/depth/modelchk Skill is valid!

Limits. This is prose: nothing here binds a tier at runtime, so whether a binder honors these rules is proven in the binder's own tests, as #7624 did with its tie and consent regressions. The six domain words are a fixed vocabulary; a task that spans two picks the one it depends on most, or general. The pull request targets feat/falsifiable-reflexes, where paperthin collects the 1.0.0 work.

- Add a neutral task_domain output and state how a binder resolves a tier against live measured capability in that domain: a user mapping outranks measurement, an unlisted model is unmeasured rather than weak, and the source's terms decide whether scores may be shown
- the tier does not depend on the order candidates are listed in, one model offered through two routes counts once, and at one price the higher measurement wins
- price rank takes over only when the source cannot be read or measures fewer than two candidates, the binder says which, and a user mapping outranks price rank too
- an unenabled source is the first reason the binder falls back to price rank
- the seven-field report has no slot for unmeasured models, so the binder, not the report, says so
- it ranks after the priced models instead of dropping out
@LilMGenius
LilMGenius force-pushed the feat/modelchk-task-domain branch from ea54a75 to 38a7f59 Compare November 2, 2026 03:48
@LilMGenius
LilMGenius changed the base branch from main to feat/falsifiable-reflexes November 2, 2026 03:48
@LilMGenius
LilMGenius merged commit 38a7f59 into feat/falsifiable-reflexes Nov 2, 2026
4 of 5 checks passed
@LilMGenius
LilMGenius deleted the feat/modelchk-task-domain branch November 2, 2026 03:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant