docs: slim CLAUDE.md and split long-form rationale into .claude - #174
Merged
Merged
Conversation
CLAUDE.md is auto-loaded into every request, so a 12,524-word changelog in it is a tax on every task. It carried a single 21,263-character bullet and a 14,025-character lint-rule catalogue. Split by purpose rather than trimming uniformly: - CLAUDE.md (12,524 -> ~2,600 words) keeps the working reference: layout, the criteria table, evaluation flow, dev commands, the lint rules that constrain routine edits, and the doc-SSOT rules. - .claude/architecture-notes.md (new) takes the long-form design rationale verbatim, under headings. It is git-tracked but not auto-loaded, so the knowledge survives without the per-request cost. The CE rule catalogue is now a pointer: every rule already carries its full rationale in its own module docstring, and a prose copy of a registry with no sensor decays silently -- which is what test_claude_md_names_every_registered_model already exists to catch. Also adds verified documentation links: every one of the 15 criteria table rows deep-links to its section in the Task Definition Guide, plus pointers to the report schema, dataset, dialog-mode, docker-isolation, CI-gate, plugin and per-agent docs. All 52 links and 25 anchors were resolved against the tree, and every anchored heading is punctuation-free so GitHub, Starlight and python-markdown agree. No docs index table was added: CE028 does not scan CLAUDE.md, so a copy here would be an unguarded fourth index. Constraints preserved deliberately: the CE030 model list stays on one physical line (the test is line-based), all six /coder-eval:* skill names, and the "six skills" count phrasing. make lint: 635 passed. make test: 5678 passed, 2 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015qJBGVKjSeTucaw5h36Pt6
uipreliga
requested review from
CarlesUIPath,
akshaylive,
bai-uipath and
tmatup
as code owners
September 14, 2026 22:39
|
Claude finished @uipreliga's task in 1m 36s —— View job Review in Progress
|
bai-uipath
approved these changes
Sep 14, 2026
`.claude/architecture-notes.md` names SKILL_SOURCE_PATH, but no CE045 glob reached it, so the coverage test failed the build. Add `.claude/**/*.md`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SB97HMVXU1NMpQcDhbYkPz
uipreliga
added a commit
that referenced
this pull request
Sep 16, 2026
Rebase fallout from #174/#177, which replaced CLAUDE.md's fenced directory tree with `ls` plus selective bullets, split `.claude/architecture-notes.md` into `.claude/notes/`, and added the `make docs-budget` prose gate. - Drop CE067. It asserted that CLAUDE.md's tree names every top-level package member; that tree no longer exists, and the bullets that replaced it are deliberately not exhaustive, so the rule's invariant is now false by design. The surviving half — no bullet may name a path that does not exist — is recorded in .claude/harness-candidates.md rather than rebuilt here. - Move the reports-package rationale into `.claude/notes/reporting.md`, and retarget `reports_stats` / `reports_junit` in notes/ to their new homes. - Bring `result_metrics.py` and `run_record.py` under the prose budget by adopting main's trimmed wording for the comments and docstrings they inherited, including its `.claude/notes/` pointers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
uipreliga
added a commit
that referenced
this pull request
Sep 16, 2026
…layer (#176) * feat(pricing): 1/5 — generate the evalboard rate table from pricing.py The frontend's rate table was a hand-copy of `_PRICING`, kept honest by five layers of bookkeeping: a regex parser that re-read pricing.py at test time, a meta-guard against that regex silently narrowing, a DELIBERATELY_UNMIRRORED exemption set, a staleness guard for the exemption set, and a comment asking the next reader to keep the set honest. It still shipped a real bug — claude-sonnet-5, gpt-5.6-sol, gpt-5.6-terra and gpt-5.6-luna sat in the exemption set under "the evalboard never runs them" while appearing ~32k / ~2k / ~17k / ~2k times in the run corpus, so those runs rendered "—" for cost with nothing failing. If a test can read the table, a generator can emit it. `make pricing-mirror` now renders evalboard/lib/pricing.generated.ts from pricing.builtin_rates(), and CE065 re-renders and diffs it. All five bookkeeping layers delete. The one thing the exemption set encoded that was NOT bookkeeping — that three OpenRouter models must stay unpriced on the frontend so runs.ts apportions the provider's real per-call bill instead of a static estimate — becomes data on the rate itself: ModelPricing.per_request_billing. The generator skips those rows, so nothing has to remember them. The field is defaulted and last, so 4-positional construction (including the out-of-tree coder_eval_uipath rate card) is unaffected; it participates in register_pricing's anti-shadow comparison, which is correct. Generating the table deliberately ADDS gpt-5.4-mini, gpt-5.4-nano, gpt-5.4-pro and gpt-5.5-pro to the frontend — the four exemption entries that were pure drift. Also corrects every surface that told a reader to hand-edit pricing.ts or named the deleted parity test: the Makefile and CI-job comments, litellm/README.md's "register in both tables" step, CLAUDE.md, and two stale evalboard consumer comments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGRvWucHgYgpcGhmyHrdBg * refactor(reports): 2/5 — two DRY fixes and hoist 18 function-local imports SLOW_PARAMS_PREVIEW_CHARS was defined in reports.py and used by reports_html.py, while the module that DEFINES it truncated with the literal 50 twice. The variant Token Usage card re-derived TokenUsage.total_tokens inline — a fourth home for arithmetic the model already owns. Both now have exactly one definition; neither changes a rendered byte at today's values, which is the point. Of the 19 function-local imports across the three report modules, 15 had their module edge at top level already, so hoisting them is adding a name to an existing line. Three add a genuinely new top-level edge (analysis in reports.py and reports_html.py, reports_html in reports_experiment.py — verified acyclic). ONE stays deferred: reports.py's `from .criteria import ...`. Importing coder_eval.criteria runs pkgutil auto-discovery with registry side effects, which hoisting would put on the path of every `import coder_eval.reports`. It now carries a comment saying so, and a subprocess test asserts coder_eval.criteria stays out of sys.modules — the decision is pinned rather than remembered. Also documents the run.json `input_tokens` seam at its writer: the key carries uncached_input_tokens, NOT the derived TokenUsage.input_tokens total, and evalboard/lib/runs.ts depends on that reading. Same word, two quantities — the name is fixed by the run.json contract and cannot change without breaking archived runs. No golden value is re-baselined: the test diff is additions only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGRvWucHgYgpcGhmyHrdBg * refactor(reports): 3/5 — split reports_stats.py into stats, result_metrics and helpers reports_stats.py had become three unrelated modules sharing a file: a distribution-free numeric core, a set of EvaluationResult metrics the ORCHESTRATOR consumes mid-run, and the report renderers' own helpers. The orchestrator importing a "reports" module for a number it needs during a run is the layering wart; the file was the reason it had to. stats.py pure statistics. Imports NOTHING from coder_eval — asserted by a test, not by convention, because that is the whole reason the numeric core can be reasoned about in isolation. result_metrics.py turn_time_buckets, visible_turn_count, has_final_reply, expected_turns_overage. Consumed by the orchestrator during a run as well as by the reporters, so it is not a report. Deliberately NOT folded into timing.py, which has no EvaluationResult dependency and is imported by every agent. reports_stats.py what is left: report-shaped helpers over variant and experiment results, plus the display formatters (fmt_mean_sd, fmt_p) — presentation, not computation. Its cycle rationale is restated rather than deleted: it is LIVE, not historical. experiment -> html -> helpers, so folding the helpers into the experiment reporter would close a cycle. Also moves eval_result_to_task_dict to run_record.py. It writes one run.json row — a run-record serializer, not a report — and its placement was the only reason orchestration/batch.py reached into the reports layer at all. Moving it is what lets Phase 4's CE066 allowlist be purely writers instead of carrying a serializer as a permanent exception. Every one of the 21 moved definitions is byte-identical to its pre-move form, verified by AST comparison. A characterization test pins the full run.json row against a snapshot captured before the move; the non-finite sensor is retargeted to coder_eval.stats and still guards the same 7 functions. Two test files that held only tests for moved names are merged into the files named for those names, rather than left behind as misnamed orphans. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGRvWucHgYgpcGhmyHrdBg * refactor(reports): 4/5 — reports/ package, durations.py, and CE066 The five reports_*.py modules become a package (markdown/html/experiment/junit/ helpers) whose __init__ re-exports the public writer surface. Private names stay private: the 12 that tests reach for are imported from their submodule, so the package's API is not a function of its test suite. `format_ms` moves to durations.py. formatting.py imports claude_agent_sdk for the payload formatters, and the reports package should not reach through an SDK-shaped module for a 14-line duration formatter. NOTE this does NOT make the package SDK-free — models/agent_config.py imports ClaudeAgentOptions and every report module needs models. The docstring and tests say what is actually true rather than what the plan hoped. CE066 pins the layering the split establishes: core may import only the package's public WRITERS. An allowlist, not a denylist, so a new report helper is banned from core by default. It checks BOTH the absolute and the relative spelling — the first draft matched only `node.module`, which for `from ..reports import X` holds "reports" with the dots in `node.level`, so it fired on neither of the two real edges in the tree and its tests passed because they used the absolute form. An unrun assertion is documentation, not enforcement. The core-layer predicate moves to a shared _layers.py that CE004 and CE066 both read, so a package added to one cannot escape the other. It is stated as "every module directly under src/coder_eval/ is core": naming only orchestrator.py left result_metrics.py exempt — the module CE066's own fix message tells you to move your metric into. Also fixes a latent packaging break: .gitignore's bare `reports/` (meant for run output) matched src/coder_eval/reports/, leaving __init__.py untracked and building a wheel with zero files under coder_eval/reports/. Anchored to /reports/. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: 5/5 — retarget every reports_* reference and record the rationale The tree, the review-command module lists and a handful of prose references still named modules that no longer exist. CLAUDE.md's `reports_html.py` line also called it "the evalboard's static twin" — a parity promise nobody was keeping: it has ~40 private renderers and nothing tests it against the 31k-line evalboard. That line goes with the entry. Adds `## The reports package` to .claude/architecture-notes.md, which is where CLAUDE.md's preamble sends a reader for rationale. It records the CE066 layering, why stats.py / result_metrics.py / run_record.py each sit outside the package, the relative-import trap CE066's first draft fell into, and — the part a future reader would otherwise re-litigate — the DECISION NOT to build a shared section-data layer, with the measurement behind it: only 1 of the 4 "duplicated" section pairs shares an input shape, and the differences in the rest are per-surface presentation, not drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(lint): CE004 never fired on the relative import spelling CE004 matched `node.module` against `^coder_eval\.cli`, but a relative import keeps its dots in `node.level` and leaves the rest in `node.module` — so `from ..cli import run_command` arrives as `level=2, module="cli"` and matched nothing. The relative form is this codebase's dominant idiom, so the rule has been guarding roughly nothing since it was written, with its tests green because they used the absolute spelling. Found while fixing the identical bug in CE066 during the reports split. Two rules independently falling into the same trap is the definition of a shared helper, so the matching moves to `_layers.imports_package` next to the core-layer predicate both rules already share — the same reasoning that put `is_core_path` there. The helper also fixes a narrower bug the regex had: `^coder_eval\.cli` prefix- matched `coder_eval.client`. Matching is now on a package boundary. Adds the regression tests both rules were missing: every spelling of a banned import, plus the prefix-bleed case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore: record two deferred harness candidates from the reports consolidation Both are real gaps with a known shape but neither is ~30 minutes of work: a hand-edit guard for generated surfaces needs a checksum gate rather than a diff, and a prose-path resolver has the same tree-parsing problem the plan already measured and declined for CLAUDE.md alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: code review fixes for the reports consolidation Three findings from the cross-phase review, all whole-diff-only: CE004 and CE066 both missed a THIRD import spelling. `imports_package` handled the absolute and relative forms, and `is_bare_package_import` handled `from . import reports` — but `from coder_eval import reports` is level=0 with module="coder_eval", which matched neither. It binds the package, so every attribute read through it was invisible to both rules. That spelling is a real in-tree idiom (`from coder_eval import __version__`), which the tests now pin as the negative case alongside the positives. reports/__init__ published nine names nothing imports. The export list was measured before packaging as "every name imported from a reports module", which at that point included the five modules importing EACH OTHER; those became intra-package `from .markdown import …` and need no re-export. 25 -> 16, and the docstring no longer claims more than it delivers. Stale cross-repo pointers the phase greps missed by stopping at the src/ boundary: two evalboard tests still cite the deleted pricing-parity test as the authority on rates (it is CE065 now), runs.ts and variants.ts still name reports_experiment.py / reports_junit.py, and a workflow comment does too. The runs.ts one matters most — it is the far half of the `input_tokens` seam Phase 2 deliberately documented at its writer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(lint): 1/4 — is_core_path covers the whole core layer `_layers.is_core_path` was a denylist of ten directory names plus a top-level-module regex, so `isolation/` — the `driver: docker` evaluation path — was invisible to BOTH CE004 and CE066. Its unanchored `[/\\]coder_eval[/\\]` also classified a repo-root file as core, because this project's own checkout directory is named `coder_eval`. Replace it with the anchored allowlist its own docstring already described: everything under `src/coder_eval/` is core except the `cli/` and `reports/` packages, so a new subpackage is core by default rather than exempt until someone notices. Uses the established `(?:^|[/\\])src[/\\]coder_eval[/\\]` spelling rather than a new variant — the defect being fixed was a regex that disagreed with its siblings. Core-set delta: +isolation/__init__.py, +isolation/docker_runner.py, +resources/__init__.py; zero removals; zero new CE004/CE066 violations. Four CE004 fixtures in tests/test_lint_runner.py built paths without a `src/` segment; three stop firing under the anchor and the fourth passes vacuously, so all four are re-anchored. `make lint` does not run that file. `TestCoreLayerMembership` pins the non-core set against the real filesystem and pins the residual `~/src/coder_eval` collision as unreachable rather than asserting it away. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(lint): 2/4 — declare CE044 and CE065 to ruff, and pin the id space `[tool.ruff.lint] external` is what stops ruff reporting RUF102 for a `# noqa: CE0xx` it does not own. It was two ids short: CE065, and CE044 — which the review missed and only prototyping the sensor found. Both are `@pytest.mark.lint` classes rather than BaseRules, and that is exactly why they slipped: `TestRuffExternalCoversEveryRule` already guarded this list, but derived the known ids from `ALL_RULES` alone, so it was blind to half the rule space it was meant to cover. Extend that class rather than adding a second one beside it: `_known()` now unions `ALL_RULES` with the `class TestCE\d{3}` ids scraped from this file, and `test_no_dead_entry_survives` asserts the other direction, so a declared id for a deleted rule fails too. Both messages name pyproject.toml and the offending ids. `tests/lint/runner.py`'s id-claiming note carried the same hand-maintained enumeration and had fallen behind CE044 identically; it now points at the grep instead of listing the ids. Nothing was red for want of these two entries — no `# noqa: CE044` or `# noqa: CE065` exists in the tree — so the fix is pre-emptive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(reports): 3/4 — cover the HTML slowest-commands truncation branch `reports/html.py:825` — the `params_preview += "..."` arm — was the one uncovered line in the renderer. Its markdown twin got both a truncation and a boundary test when SLOW_PARAMS_PREVIEW_CHARS was introduced; the HTML side got neither. Adds the twin pair. Both read the constant rather than the literal 50, so they survive a change to it, and both were mutation-checked: flipping `>` to `>=` fails the boundary case, deleting the ellipsis arm fails the truncation case. `str(dict)` emits single quotes that `_esc` renders as `'`, so the raw cell is longer than the preview. The assertion unescapes before measuring; the renderer's escaping is untouched. No source change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: 4/4 — retarget prose that names deleted report modules The consolidation's own greps could not reach these: nine present-tense references in tests/ naming `reports.py`, `reports_html.py`, `reports_experiment.py` or `reports_junit.py` — modules that no longer exist — plus CE053's fixture default, whose `: str = ` spacing hid it from a `filepath="src/…"` grep. Classified rather than sed'd: eight further sites keep their wording because they date a past incident in past tense, name a test file that still exists, or are the deliberate `reports_html` local alias. All eight verified unedited. Also deletes CLAUDE.md's `optimize/` tree entry. The directory is absent from HEAD and from main; it lives only on the unmerged `feat/plugin-optimize-skill` branch, so the tree as documented did not match the tree as shipped. No source file changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: code review fixes for the reports-consolidation review fixes Two Medium findings, each flagged independently by a reviewer: - `_layers.py` spelled the 30-char package prefix twice, in the module whose own docstring argues a second copy of a definition is how the two drift. `_NON_CORE` now derives from `_PKG.pattern`, so widening one cannot leave the other behind — a divergence no test could have seen, because both assertions only ever feed src-layout paths. Compiled pattern unchanged. - The trailing `[/\\]` in `(cli|reports)[/\\]` is what keeps a top-level module whose name merely STARTS with `reports` or `cli` core. Nothing pinned it: deleting the separator left all 738 tests green. Pins added for `reports_legacy.py` and `cli_helpers.py`; both fail under that mutation. Also, from the same reviews: - Pin the reachability argument that made the `~/src/coder_eval` residual safe. It lived only as docstring prose; adding CE004 or CE066 to `_ALSO_SCAN_TESTS` would hand them a whole `tests/` tree that matches `_PKG` on an ordinary clone layout. Now one assertion. - `tests/lint/runner.py`'s new grep told the next author to run `^class Test(CE\d{3})`. GNU and BSD `grep -E` read `\d` as a literal `d` and report zero hits — "no ids taken", the exact miss the note exists to prevent. Respelled `[0-9]{3}`. - CLAUDE.md's tree audit ran one way only. Deleting the phantom `optimize/` entry was right, but `errors/` and `plugins.py` exist and were absent — and `plugins.py` is the SPI the "Adding a New Agent" section points at. - Record `ce048`'s near-variant of the shared path regex in the candidates entry. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(lint): CE067 — CLAUDE.md's tree must name every top-level package member Harness loop, closed in-session. The directory tree is the map an assistant reads before touching this package, and it drifted in both directions across two consecutive plans while only one direction was ever audited: a phantom `optimize/` row survived for a directory that lives solely on an unmerged branch, and the sweep that removed it walked entries -> filesystem, so it could not see that `errors/` and `plugins.py` were missing — the second being the plugin SPI that "Adding a New Agent" tells you to use. Asserts both directions over the top-level rows only; nested rows stay illustrative, so a new sibling module is not a forced docs edit. Proven to fire each way: re-adding the `optimize/` row and deleting the `plugins.py` row each fail with the offending name. Writing it also found a third omission the hand audit missed — `__init__.py`, here deliberately ignored along with the build and typing markers. Claims id 067; the Phase 2 parity sensor required the pyproject entry immediately, which is the sensor doing its job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(lint): drop a personal path and de-duplicate the isolation pin No test datum should carry a developer's home directory. The repo-root case now uses `/home/dev/src/exp/coder_eval/conftest.py`, which keeps the shape that mattered — a `src` component that is NOT the package's parent, so it exercises the anchor rather than merely the absence of `src`. `ISOLATION` and a verbatim three-line docstring were copied into both the CE004 and CE066 test classes. Hoisted to one module-level `CORE_ISOLATION` with the rationale stated once, so the two pins cannot drift to different paths. Verified the pins still bite: re-exempting `isolation/` in `_NON_CORE` fails four tests. Also renames `test_ce008_skips_files_outside_scope` to `ce009` — it sits in the CE009 block and exercises `YamlModelsForbidExtras` (CE009), while CE008's own tests cover `ReadTextExplicitEncoding`. Pre-existing mislabel, flagged in review. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(lint): drop two function-local json imports that shadow the module one `tests/test_custom_lint.py` imports `json` at module level, and two tests re-imported it inside the function body. CodeQL flagged the one this PR added (in the pricing-mirror test); the other, in the activation-rows test, predates the PR and has the same shape, so both go. Ruff has no rule for a repeated import inside a function body, which is why `make check` never saw either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(lint): CE004 checks reports/ — stop borrowing CE066's exemption set CE004 and CE066 shared one predicate, `is_core_path`, whose exemption set is `{cli, reports}`. That set is CE066's: the reports package may reach into itself. CE004's is only `{cli}` — the reports package runs without the CLI (the orchestrator writes a task report mid-run), so a `cli` import added inside `reports/` closes a cli -> orchestration -> reports -> cli cycle, and CE004 would have stayed silent. Latent, not live: widening the scope finds 0 violations. The rules still share what must not drift — the anchored package regex and the `cli/` boundary, now `is_package_path` and `is_cli_path` — but each states its own exemptions. CE066 keeps `is_core_path`; CE004's scope is the package minus `cli/`. Tests, each mutation-checked against CE004 going back to the core predicate: - `test_the_reports_package_is_in_scope` — a `cli` import in reports/ violates. - `TestCoreLayerMembership.test_ce004_scope_is_every_module_outside_cli` runs the RULE at every real module path. Its first draft recomputed the scope from the helpers and passed under that mutation, so it now calls the rule. - `test_the_reports_package_itself_stays_exempt` pins that CE066's scope did not widen with CE004's. Closes the deferred harness candidate; architecture notes updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(lint): resolve a relative import against the importing file CE004 and CE066 matched `node.module` against the bare package name whenever `node.level` was non-zero, so `from .reports import x` inside `orchestration/` — i.e. `coder_eval.orchestration.reports` — read as the reports layer, and `from ...reports import x` from a sub-package read as it too although it escapes `coder_eval` entirely. Latent: nothing in the tree is nested that way today. `_absolute_module` now resolves the dots against the file's own package, so both rules compare one fully-qualified name in either spelling. The CE004 scope probe moves to the absolute spelling: a relative one resolves against the importing file, so `..cli` names the cli layer only from inside a sub-package and that test would measure depth instead of scope. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix: reconcile the reports split with main's docs restructure Rebase fallout from #174/#177, which replaced CLAUDE.md's fenced directory tree with `ls` plus selective bullets, split `.claude/architecture-notes.md` into `.claude/notes/`, and added the `make docs-budget` prose gate. - Drop CE067. It asserted that CLAUDE.md's tree names every top-level package member; that tree no longer exists, and the bullets that replaced it are deliberately not exhaustive, so the rule's invariant is now false by design. The surviving half — no bullet may name a path that does not exist — is recorded in .claude/harness-candidates.md rather than rebuilt here. - Move the reports-package rationale into `.claude/notes/reporting.md`, and retarget `reports_stats` / `reports_junit` in notes/ to their new homes. - Bring `result_metrics.py` and `run_record.py` under the prose budget by adopting main's trimmed wording for the comments and docstrings they inherited, including its `.claude/notes/` pointers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(pricing): carry the exemption set into the generated mirror The generator kept only `per_request_billing` as an exemption axis, so it reproduced three of the hand-copy's seven exclusions and silently priced the other four. Those four were not drift: `gpt-5.4-mini`, `gpt-5.4-nano`, `gpt-5.4-pro` and `gpt-5.5-pro` were enumerated exemptions in the parity test this branch deletes. Pricing them is a behaviour change smuggled in as a refactor — latent only because no experiment config references them today. The exemption set encoded two different claims and they now survive separately. That a routed model MUST NOT be priced is a fact about the rate, and stays on `ModelPricing.per_request_billing`. That a heavy frontier variant is not worth pricing on the board is a fact about the FRONTEND, so it is `DELIBERATELY_UNMIRRORED` beside the generator. What the hand-copy got wrong was not having an exemption set but letting it go stale unnoticed, so the deleted test's staleness guard comes back as `_assert_exemptions_are_live`: an id that has left `pricing.py` fails `make pricing-mirror` and CE065 rather than sitting there silencing nothing. The generated table now reproduces the hand-copy's 52 keys exactly — no additions, no removals. CE065 asserts both axes from their declared sources, and the evalboard's consumption guard pins the four as unpriced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
CLAUDE.mdis auto-loaded into every request, so a 12,524-word changelog in it is a tax on every task. It carried a single 21,263-character bullet and a 14,025-character lint-rule catalogue.The prose quality was never the problem — the location was.
What
Split by purpose rather than trimming uniformly:
CLAUDE.md(12,524 → ~2,600 words).claude/architecture-notes.md(new)The CE rule catalogue is now a pointer. Every rule already carries its full rationale in its own module docstring, and a prose copy of a registry with no sensor decays silently — which is exactly what
test_claude_md_names_every_registered_modelalready exists to catch.Documentation links
Every one of the 15 criteria table rows now deep-links to its section in the Task Definition Guide, plus pointers to the report schema, dataset, dialog-mode, docker-isolation, CI-gate, plugin and per-agent docs.
stop_early,API Routing & Benchmarking) are deliberately linked at file level.CLAUDE.md, so a copy here would be an unguarded fourth index that decays.Constraints preserved deliberately
/coder-eval:*skill names (SKILL_DOC_SURFACES).setting_sources: []guidance, which a line-by-line diff caught as landing in neither file on the first pass.Verification
make lint— 635 passedmake test— 5,678 passed, 2 skipped (both environmental: needs a realANTHROPIC_API_KEY; Windows-only)Not in this PR
A proposal to replace the Design Principles values (DRY/YAGNI/KISS) with seven checkable "Never ship" rules, three
PreToolUse/Stophooks, and an Opus-in-tests lint rule. Discussed but not applied.🤖 Generated with Claude Code
https://claude.ai/code/session_015qJBGVKjSeTucaw5h36Pt6