Skill registries tell you what's popular. skillfit tells you what actually works.
Measure whether a skill, rules file, or MCP setup improves your agent on your tasks —
then install only what survives the experiment.
The agent-config ecosystem has solved distribution (npx skills add, plugin marketplaces, MCP registries) but not selection. The evidence says unverified configuration can hurt:
- Curated skills improve pass rates by +16.6pp on average — but agent-self-generated skills score −1.3pp, and focused skills beat large bundles (SkillsBench, arXiv:2602.12670)
- LLM-generated context files scored −3% while raising inference cost by 20%+ (ETH Zurich, arXiv:2602.11988)
- 36% of scanned public skills contain prompt injection (Snyk ToxicSkills, 2026-02)
skillfit is the missing measurement layer: paired A/B experiments with deterministic verifiers, statistical verdicts, and trigger-rate measurement — packaged as a CLI anyone can run.
A real run, measuring whether an agent even bothers to load a skill when it is installed but not mentioned:
$ npx skillfit eval ./skills/code-review --mode trigger --bench code-review --agent kimi-code
TASK FIRE? FIRED UNKNOWN ERRORS PASS
review-r1 yes 1/3 0 0 3/3
review-r2 yes 0/3 0 0 3/3
review-r3 yes 0/3 0 0 3/3
explain-x1 no 0/3 0 0 3/3
Trigger recall : 1/9 (11%) [95% CI 2%–44%]
False-trigger rate : 0/3 (0%) [95% CI 0%–56%]
Precision : 1/1 (100%) [95% CI 21%–100%]
F1 : 0.20 (no CI: a harmonic mean of two proportions has no closed-form binomial interval)The skill fired once in nine in-domain tasks — and the tasks pass 3/3 without it. That is a verdict no registry can give you.
Captured 2026-09-22 on Kimi Code against the four-task code-review bench. The bench has since gained a fifth task; the four metric lines above are re-rendered from that run's recorded per-task counts, so the run is real and the formatting is current.
# 1. Health-check your current setup (read-only, safe)
npx skillfit doctor
# 2. A/B-test a skill before installing it (pick a bundled bench by name, or pass your own path;
# use --agent to drive a local agent CLI instead of an API key)
npx skillfit eval ~/.agents/skills/some-skill --bench code-review --trials 3
# 2b. Or measure whether the agent triggers the skill on its own (and only when it should)
npx skillfit eval ~/.agents/skills/some-skill --mode trigger --bench code-review --agent kimi-code
# 3. Install only the evidence-backed minimal set (dry-run by default)
npx skillfit install
# Optional: teach your agent to drive it (copy the driver skill into your agents dir)
cp -r skills/skillfit ~/.agents/skills/Supported agents: Claude Code, OpenAI Codex CLI, Kimi Code (capability matrix — machine-readable, dated, doc-linked). Trigger-mode capture is currently verified for Kimi Code and Codex CLI.
| Command | What it does | Writes? |
|---|---|---|
doctor |
Detects installed agents, checks rules bloat, skill validity/conflicts, MCP config parseability, silent-failure traps (e.g. AGENTS.md that Claude Code never reads) | Never |
report |
Skill usage receipts from local session history: fires per skill per agent, and the never-fired list (the pure routing/context tax) | Never |
eval <skill> |
Default (--mode inject): paired baseline/treatment runs, deterministic verifier + optional blind LLM judge, token-cost delta, verdicts via McNemar exact test + paired bootstrap CI, plus graded facet-score CIs when the bench emits checks. --mode trigger: installs the skill instead of injecting it and measures trigger recall / false-trigger rate from the agent transcript |
runs/ locally |
bench |
init scaffolds a bench directory with a working example task; check validates a bench offline (verifier self-tests, oracle/NOP gates, mock-arm probes, fixture hygiene, trigger-label coverage); add --freeze turns a failure you just watched into a permanent bench task, and --decompose has an agent draft the verifier + oracle, admitted only if both gates pass |
init/add after confirmation; check never |
install |
Managed-block rules (<!-- SKILLFIT_START/END -->, idempotent), skill copy with conflict protection, commit-pinned lockfile, post-install verification. Writes are staged then renamed, so a failure part-way applies nothing; your original is kept at <file>.skillfit-bak and the first backup wins, so later updates cannot overwrite it. --dry-run reports conflicts and exits 0; add --strict to make them fail (CI gates) |
Only after confirmation |
Evals are only as good as their tasks. A bench is just a directory — bench.json + fixtures + a deterministic verifier. Scaffold one with npx skillfit bench init, freeze a real failure you just watched your agent botch with npx skillfit bench add <bench> --freeze, validate offline with npx skillfit bench check, and model it on your own production scenarios: benches/README.md. If you have never written one, start with docs/bench-authoring.md — it walks the seven steps end to end on a single real task, and covers the ways a bench produces confident wrong numbers.
We ran skillfit's harness on 8 popular workflow skills (24 baseline/treatment pairs, Codex CLI, frozen 2026-07 baseline). Only 1 of 8 showed a repeatable benefit:
| Skill | Quality Δ | Input tokens | Verdict |
|---|---|---|---|
diagnosing-bugs |
+66.7pp test-asset gain (2/3 runs) | +11.7% | Conditional — hard bugs only |
code-review |
+3.3pp (unstable) | +9.1% | Not enough evidence |
tdd |
0.00pp | +9.2% | No measurable gain |
doubt-driven-development |
0.00pp | +20.7% | No measurable gain |
security-and-hardening |
0.00pp | +27.4% | No measurable gain, highest cost |
| 3 more | 0.00pp | +14.7~17.0% | No measurable gain |
Full methodology and raw manifests: evidence/. Reproduce it yourself with skillfit eval.
- Standards, not formats. AGENTS.md (AAIF), SKILL.md,
.agents/skills/,.mcpb— we write what agents already read. - Deny by default. We install only what a profile explicitly declares, pinned by content hash.
- Dry-run first. Every write command prints its plan before touching a file. Backups always.
- Honest numbers. Every claim links to a manifest with model version, skill hash, date, and variance. Verdict semantics are frozen in docs/metrics.md: significance comes from an exact McNemar test over discordant pairs, deltas carry paired-bootstrap CIs, and underpowered runs are labeled indicative, never "effective".
Not affiliated with Anthropic, OpenAI, Moonshot AI, or any agent vendor. Evaluation results depend on model version, harness, and tasks — treat them as dated evidence, not eternal truth.
- doctor / eval / install core loop
- Paired A/B harness with blind judging
- Statistical verdicts (McNemar exact + paired bootstrap CI, manifest v2)
- Trigger-rate measurement (
--mode trigger: recall / false-trigger rate with Wilson CIs) - Bench scaffolding (
bench init+bench check), failure freezing (--freeze), git-history mining (--from-commit), difficulty calibration (--calibrate) - Trigger capture for Claude Code (blocked: needs working auth)
- Community bench & evidence submissions (reproducible-config CI re-runs, not trust-me results)
- Cursor / Gemini CLI / OpenCode adapters
- MCP server config evaluation
See CONTRIBUTING.md. The highest-value contribution is a bench built from your real workflow. Release notes live in CHANGELOG.md; security reports go to SECURITY.md.