Repository navigation
feat(modelchk): name the task domain a tier is measured in - #24
Merged
Merged
Conversation
- Add a neutral task_domain output and state how a binder resolves a tier against live measured capability in that domain: a user mapping outranks measurement, an unlisted model is unmeasured rather than weak, and the source's terms decide whether scores may be shown
- the tier does not depend on the order candidates are listed in, one model offered through two routes counts once, and at one price the higher measurement wins
- price rank takes over only when the source cannot be read or measures fewer than two candidates, the binder says which, and a user mapping outranks price rank too
- an unenabled source is the first reason the binder falls back to price rank
- the seven-field report has no slot for unmeasured models, so the binder, not the report, says so
- it ranks after the priced models instead of dropping out
LilMGenius
force-pushed
the
feat/modelchk-task-domain
branch
from
November 2, 2026 03:48
ea54a75 to
38a7f59
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
modelchk sized a tier and an effort but never said what a tier is measured against, so a binder mapping tiers to models had only price to go on. When two vendors price their lineups the same, price rank picks whichever model is listed first, and every role drifts to one vendor. Role auto-assign in opencodex did exactly that until lidge-jun/opencodex#7624 tiered candidates by measured capability.
Failing input. Sizing any task with modelchk returns
recommended_tierandrecommended_effortand nothing that tells a binder which measured strength the tier depends on. A binder given two equally priced candidates, one strong at coding and one at writing, has no basis in the skill to prefer either for a coding task.Cause.
skills/depth/modelchk/SKILL.mddefined a tier only as "the cheapest sufficient class" and stopped at the two dials. It said nothing about the capability axis a tier is measured on or how a binder turns a tier into a model.Change.
task_domain(coding,agentic,reasoning,writing,knowledge,general). It sizes nothing; it names the measured strength the tier depends on. A workflow step picks it, the report carries it, and the verification checklist requires exactly one.The tie, fallback, opt-in and unpriced sentences come from defects reviewers found in #7624 before it merged: equal scores split across tiers by provider order, a fallback described more narrowly than the code ran it, an unpriced measured model documented as never proposed, and a benchmark download the owner ruled must be opt-in.
Verification (on this branch at the head of this PR):
node scripts/gates/llms.cjsllms-full.txtregeneratednpm run checknpm testuv run --with pyyaml python <skill-creator>/scripts/quick_validate.py skills/depth/modelchkLimits. This is prose: nothing here binds a tier at runtime, so whether a binder honors these rules is proven in the binder's own tests, as #7624 did with its tie and consent regressions. The six domain words are a fixed vocabulary; a task that spans two picks the one it depends on most, or
general. The pull request targetsfeat/falsifiable-reflexes, where paperthin collects the 1.0.0 work.