Conversation
The frontend's rate table was a hand-copy of `_PRICING`, kept honest by five layers of bookkeeping: a regex parser that re-read pricing.py at test time, a meta-guard against that regex silently narrowing, a DELIBERATELY_UNMIRRORED exemption set, a staleness guard for the exemption set, and a comment asking the next reader to keep the set honest. It still shipped a real bug — claude-sonnet-5, gpt-5.6-sol, gpt-5.6-terra and gpt-5.6-luna sat in the exemption set under "the evalboard never runs them" while appearing ~32k / ~2k / ~17k / ~2k times in the run corpus, so those runs rendered "—" for cost with nothing failing. If a test can read the table, a generator can emit it. `make pricing-mirror` now renders evalboard/lib/pricing.generated.ts from pricing.builtin_rates(), and CE065 re-renders and diffs it. All five bookkeeping layers delete. The one thing the exemption set encoded that was NOT bookkeeping — that three OpenRouter models must stay unpriced on the frontend so runs.ts apportions the provider's real per-call bill instead of a static estimate — becomes data on the rate itself: ModelPricing.per_request_billing. The generator skips those rows, so nothing has to remember them. The field is defaulted and last, so 4-positional construction (including the out-of-tree coder_eval_uipath rate card) is unaffected; it participates in register_pricing's anti-shadow comparison, which is correct. Generating the table deliberately ADDS gpt-5.4-mini, gpt-5.4-nano, gpt-5.4-pro and gpt-5.5-pro to the frontend — the four exemption entries that were pure drift. Also corrects every surface that told a reader to hand-edit pricing.ts or named the deleted parity test: the Makefile and CI-job comments, litellm/README.md's "register in both tables" step, CLAUDE.md, and two stale evalboard consumer comments. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGRvWucHgYgpcGhmyHrdBg
…ports SLOW_PARAMS_PREVIEW_CHARS was defined in reports.py and used by reports_html.py, while the module that DEFINES it truncated with the literal 50 twice. The variant Token Usage card re-derived TokenUsage.total_tokens inline — a fourth home for arithmetic the model already owns. Both now have exactly one definition; neither changes a rendered byte at today's values, which is the point. Of the 19 function-local imports across the three report modules, 15 had their module edge at top level already, so hoisting them is adding a name to an existing line. Three add a genuinely new top-level edge (analysis in reports.py and reports_html.py, reports_html in reports_experiment.py — verified acyclic). ONE stays deferred: reports.py's `from .criteria import ...`. Importing coder_eval.criteria runs pkgutil auto-discovery with registry side effects, which hoisting would put on the path of every `import coder_eval.reports`. It now carries a comment saying so, and a subprocess test asserts coder_eval.criteria stays out of sys.modules — the decision is pinned rather than remembered. Also documents the run.json `input_tokens` seam at its writer: the key carries uncached_input_tokens, NOT the derived TokenUsage.input_tokens total, and evalboard/lib/runs.ts depends on that reading. Same word, two quantities — the name is fixed by the run.json contract and cannot change without breaking archived runs. No golden value is re-baselined: the test diff is additions only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SGRvWucHgYgpcGhmyHrdBg
…trics and helpers
reports_stats.py had become three unrelated modules sharing a file: a
distribution-free numeric core, a set of EvaluationResult metrics the
ORCHESTRATOR consumes mid-run, and the report renderers' own helpers. The
orchestrator importing a "reports" module for a number it needs during a run is
the layering wart; the file was the reason it had to.
stats.py pure statistics. Imports NOTHING from coder_eval — asserted
by a test, not by convention, because that is the whole
reason the numeric core can be reasoned about in isolation.
result_metrics.py turn_time_buckets, visible_turn_count, has_final_reply,
expected_turns_overage. Consumed by the orchestrator during
a run as well as by the reporters, so it is not a report.
Deliberately NOT folded into timing.py, which has no
EvaluationResult dependency and is imported by every agent.
reports_stats.py what is left: report-shaped helpers over variant and
experiment results, plus the display formatters
(fmt_mean_sd, fmt_p) — presentation, not computation.
Its cycle rationale is restated rather than deleted: it is LIVE, not historical.
experiment -> html -> helpers, so folding the helpers into the experiment
reporter would close a cycle.
Also moves eval_result_to_task_dict to run_record.py. It writes one run.json row
— a run-record serializer, not a report — and its placement was the only reason
orchestration/batch.py reached into the reports layer at all. Moving it is what
lets Phase 4's CE066 allowlist be purely writers instead of carrying a serializer
as a permanent exception.
Every one of the 21 moved definitions is byte-identical to its pre-move form,
verified by AST comparison. A characterization test pins the full run.json row
against a snapshot captured before the move; the non-finite sensor is retargeted
to coder_eval.stats and still guards the same 7 functions.
Two test files that held only tests for moved names are merged into the files
named for those names, rather than left behind as misnamed orphans.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SGRvWucHgYgpcGhmyHrdBg
The five reports_*.py modules become a package (markdown/html/experiment/junit/ helpers) whose __init__ re-exports the public writer surface. Private names stay private: the 12 that tests reach for are imported from their submodule, so the package's API is not a function of its test suite. `format_ms` moves to durations.py. formatting.py imports claude_agent_sdk for the payload formatters, and the reports package should not reach through an SDK-shaped module for a 14-line duration formatter. NOTE this does NOT make the package SDK-free — models/agent_config.py imports ClaudeAgentOptions and every report module needs models. The docstring and tests say what is actually true rather than what the plan hoped. CE066 pins the layering the split establishes: core may import only the package's public WRITERS. An allowlist, not a denylist, so a new report helper is banned from core by default. It checks BOTH the absolute and the relative spelling — the first draft matched only `node.module`, which for `from ..reports import X` holds "reports" with the dots in `node.level`, so it fired on neither of the two real edges in the tree and its tests passed because they used the absolute form. An unrun assertion is documentation, not enforcement. The core-layer predicate moves to a shared _layers.py that CE004 and CE066 both read, so a package added to one cannot escape the other. It is stated as "every module directly under src/coder_eval/ is core": naming only orchestrator.py left result_metrics.py exempt — the module CE066's own fix message tells you to move your metric into. Also fixes a latent packaging break: .gitignore's bare `reports/` (meant for run output) matched src/coder_eval/reports/, leaving __init__.py untracked and building a wheel with zero files under coder_eval/reports/. Anchored to /reports/. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tree, the review-command module lists and a handful of prose references still named modules that no longer exist. CLAUDE.md's `reports_html.py` line also called it "the evalboard's static twin" — a parity promise nobody was keeping: it has ~40 private renderers and nothing tests it against the 31k-line evalboard. That line goes with the entry. Adds `## The reports package` to .claude/architecture-notes.md, which is where CLAUDE.md's preamble sends a reader for rationale. It records the CE066 layering, why stats.py / result_metrics.py / run_record.py each sit outside the package, the relative-import trap CE066's first draft fell into, and — the part a future reader would otherwise re-litigate — the DECISION NOT to build a shared section-data layer, with the measurement behind it: only 1 of the 4 "duplicated" section pairs shares an input shape, and the differences in the rest are per-surface presentation, not drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CE004 matched `node.module` against `^coder_eval\.cli`, but a relative import keeps its dots in `node.level` and leaves the rest in `node.module` — so `from ..cli import run_command` arrives as `level=2, module="cli"` and matched nothing. The relative form is this codebase's dominant idiom, so the rule has been guarding roughly nothing since it was written, with its tests green because they used the absolute spelling. Found while fixing the identical bug in CE066 during the reports split. Two rules independently falling into the same trap is the definition of a shared helper, so the matching moves to `_layers.imports_package` next to the core-layer predicate both rules already share — the same reasoning that put `is_core_path` there. The helper also fixes a narrower bug the regex had: `^coder_eval\.cli` prefix- matched `coder_eval.client`. Matching is now on a package boundary. Adds the regression tests both rules were missing: every spelling of a banned import, plus the prefix-bleed case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…idation Both are real gaps with a known shape but neither is ~30 minutes of work: a hand-edit guard for generated surfaces needs a checksum gate rather than a diff, and a prose-path resolver has the same tree-parsing problem the plan already measured and declined for CLAUDE.md alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three findings from the cross-phase review, all whole-diff-only: CE004 and CE066 both missed a THIRD import spelling. `imports_package` handled the absolute and relative forms, and `is_bare_package_import` handled `from . import reports` — but `from coder_eval import reports` is level=0 with module="coder_eval", which matched neither. It binds the package, so every attribute read through it was invisible to both rules. That spelling is a real in-tree idiom (`from coder_eval import __version__`), which the tests now pin as the negative case alongside the positives. reports/__init__ published nine names nothing imports. The export list was measured before packaging as "every name imported from a reports module", which at that point included the five modules importing EACH OTHER; those became intra-package `from .markdown import …` and need no re-export. 25 -> 16, and the docstring no longer claims more than it delivers. Stale cross-repo pointers the phase greps missed by stopping at the src/ boundary: two evalboard tests still cite the deleted pricing-parity test as the authority on rates (it is CE065 now), runs.ts and variants.ts still name reports_experiment.py / reports_junit.py, and a workflow comment does too. The runs.ts one matters most — it is the far half of the `input_tokens` seam Phase 2 deliberately documented at its writer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
uipreliga
requested review from
CarlesUIPath,
akshaylive,
bai-uipath and
tmatup
as code owners
September 15, 2026 02:25
|
Claude finished @uipreliga's task in 1m 18s —— View job Code Review in Progress
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
`_layers.is_core_path` was a denylist of ten directory names plus a top-level-module regex, so `isolation/` — the `driver: docker` evaluation path — was invisible to BOTH CE004 and CE066. Its unanchored `[/\\]coder_eval[/\\]` also classified a repo-root file as core, because this project's own checkout directory is named `coder_eval`. Replace it with the anchored allowlist its own docstring already described: everything under `src/coder_eval/` is core except the `cli/` and `reports/` packages, so a new subpackage is core by default rather than exempt until someone notices. Uses the established `(?:^|[/\\])src[/\\]coder_eval[/\\]` spelling rather than a new variant — the defect being fixed was a regex that disagreed with its siblings. Core-set delta: +isolation/__init__.py, +isolation/docker_runner.py, +resources/__init__.py; zero removals; zero new CE004/CE066 violations. Four CE004 fixtures in tests/test_lint_runner.py built paths without a `src/` segment; three stop firing under the anchor and the fourth passes vacuously, so all four are re-anchored. `make lint` does not run that file. `TestCoreLayerMembership` pins the non-core set against the real filesystem and pins the residual `~/src/coder_eval` collision as unreachable rather than asserting it away. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`[tool.ruff.lint] external` is what stops ruff reporting RUF102 for a
`# noqa: CE0xx` it does not own. It was two ids short: CE065, and CE044 —
which the review missed and only prototyping the sensor found.
Both are `@pytest.mark.lint` classes rather than BaseRules, and that is
exactly why they slipped: `TestRuffExternalCoversEveryRule` already guarded
this list, but derived the known ids from `ALL_RULES` alone, so it was blind
to half the rule space it was meant to cover.
Extend that class rather than adding a second one beside it: `_known()` now
unions `ALL_RULES` with the `class TestCE\d{3}` ids scraped from this file,
and `test_no_dead_entry_survives` asserts the other direction, so a declared
id for a deleted rule fails too. Both messages name pyproject.toml and the
offending ids.
`tests/lint/runner.py`'s id-claiming note carried the same hand-maintained
enumeration and had fallen behind CE044 identically; it now points at the
grep instead of listing the ids.
Nothing was red for want of these two entries — no `# noqa: CE044` or
`# noqa: CE065` exists in the tree — so the fix is pre-emptive.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`reports/html.py:825` — the `params_preview += "..."` arm — was the one uncovered line in the renderer. Its markdown twin got both a truncation and a boundary test when SLOW_PARAMS_PREVIEW_CHARS was introduced; the HTML side got neither. Adds the twin pair. Both read the constant rather than the literal 50, so they survive a change to it, and both were mutation-checked: flipping `>` to `>=` fails the boundary case, deleting the ellipsis arm fails the truncation case. `str(dict)` emits single quotes that `_esc` renders as `'`, so the raw cell is longer than the preview. The assertion unescapes before measuring; the renderer's escaping is untouched. No source change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The consolidation's own greps could not reach these: nine present-tense references in tests/ naming `reports.py`, `reports_html.py`, `reports_experiment.py` or `reports_junit.py` — modules that no longer exist — plus CE053's fixture default, whose `: str = ` spacing hid it from a `filepath="src/…"` grep. Classified rather than sed'd: eight further sites keep their wording because they date a past incident in past tense, name a test file that still exists, or are the deliberate `reports_html` local alias. All eight verified unedited. Also deletes CLAUDE.md's `optimize/` tree entry. The directory is absent from HEAD and from main; it lives only on the unmerged `feat/plugin-optimize-skill` branch, so the tree as documented did not match the tree as shipped. No source file changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two Medium findings, each flagged independently by a reviewer:
- `_layers.py` spelled the 30-char package prefix twice, in the module whose
own docstring argues a second copy of a definition is how the two drift.
`_NON_CORE` now derives from `_PKG.pattern`, so widening one cannot leave
the other behind — a divergence no test could have seen, because both
assertions only ever feed src-layout paths. Compiled pattern unchanged.
- The trailing `[/\\]` in `(cli|reports)[/\\]` is what keeps a top-level
module whose name merely STARTS with `reports` or `cli` core. Nothing
pinned it: deleting the separator left all 738 tests green. Pins added for
`reports_legacy.py` and `cli_helpers.py`; both fail under that mutation.
Also, from the same reviews:
- Pin the reachability argument that made the `~/src/coder_eval` residual
safe. It lived only as docstring prose; adding CE004 or CE066 to
`_ALSO_SCAN_TESTS` would hand them a whole `tests/` tree that matches
`_PKG` on an ordinary clone layout. Now one assertion.
- `tests/lint/runner.py`'s new grep told the next author to run
`^class Test(CE\d{3})`. GNU and BSD `grep -E` read `\d` as a literal `d`
and report zero hits — "no ids taken", the exact miss the note exists to
prevent. Respelled `[0-9]{3}`.
- CLAUDE.md's tree audit ran one way only. Deleting the phantom `optimize/`
entry was right, but `errors/` and `plugins.py` exist and were absent —
and `plugins.py` is the SPI the "Adding a New Agent" section points at.
- Record `ce048`'s near-variant of the shared path regex in the candidates
entry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e member Harness loop, closed in-session. The directory tree is the map an assistant reads before touching this package, and it drifted in both directions across two consecutive plans while only one direction was ever audited: a phantom `optimize/` row survived for a directory that lives solely on an unmerged branch, and the sweep that removed it walked entries -> filesystem, so it could not see that `errors/` and `plugins.py` were missing — the second being the plugin SPI that "Adding a New Agent" tells you to use. Asserts both directions over the top-level rows only; nested rows stay illustrative, so a new sibling module is not a forced docs edit. Proven to fire each way: re-adding the `optimize/` row and deleting the `plugins.py` row each fail with the offending name. Writing it also found a third omission the hand audit missed — `__init__.py`, here deliberately ignored along with the build and typing markers. Claims id 067; the Phase 2 parity sensor required the pyproject entry immediately, which is the sensor doing its job. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No test datum should carry a developer's home directory. The repo-root case now uses `/home/dev/src/exp/coder_eval/conftest.py`, which keeps the shape that mattered — a `src` component that is NOT the package's parent, so it exercises the anchor rather than merely the absence of `src`. `ISOLATION` and a verbatim three-line docstring were copied into both the CE004 and CE066 test classes. Hoisted to one module-level `CORE_ISOLATION` with the rationale stated once, so the two pins cannot drift to different paths. Verified the pins still bite: re-exempting `isolation/` in `_NON_CORE` fails four tests. Also renames `test_ce008_skips_files_outside_scope` to `ce009` — it sits in the CE009 block and exercises `YamlModelsForbidExtras` (CE009), while CE008's own tests cover `ReadTextExplicitEncoding`. Pre-existing mislabel, flagged in review. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…le one `tests/test_custom_lint.py` imports `json` at module level, and two tests re-imported it inside the function body. CodeQL flagged the one this PR added (in the pricing-mirror test); the other, in the activation-rows test, predates the PR and has the same shape, so both go. Ruff has no rule for a repeated import inside a function body, which is why `make check` never saw either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CE004 and CE066 shared one predicate, `is_core_path`, whose exemption set is
`{cli, reports}`. That set is CE066's: the reports package may reach into
itself. CE004's is only `{cli}` — the reports package runs without the CLI
(the orchestrator writes a task report mid-run), so a `cli` import added
inside `reports/` closes a cli -> orchestration -> reports -> cli cycle, and
CE004 would have stayed silent. Latent, not live: widening the scope finds
0 violations.
The rules still share what must not drift — the anchored package regex and
the `cli/` boundary, now `is_package_path` and `is_cli_path` — but each
states its own exemptions. CE066 keeps `is_core_path`; CE004's scope is the
package minus `cli/`.
Tests, each mutation-checked against CE004 going back to the core predicate:
- `test_the_reports_package_is_in_scope` — a `cli` import in reports/ violates.
- `TestCoreLayerMembership.test_ce004_scope_is_every_module_outside_cli` runs
the RULE at every real module path. Its first draft recomputed the scope
from the helpers and passed under that mutation, so it now calls the rule.
- `test_the_reports_package_itself_stays_exempt` pins that CE066's scope did
not widen with CE004's.
Closes the deferred harness candidate; architecture notes updated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
@claude please review this PR in full, at head |
|
Claude finished @uipreliga's task in 1m 28s —— View job Code Review in Progress
|
bai-uipath
approved these changes
Sep 15, 2026
bai-uipath
left a comment
bai-uipath
left a comment
Collaborator
There was a problem hiding this comment.
Fix what you agree with, otherwise lgtm. The split itself is clean, and most of the +3187 is relocation rather than new code. The layering work is the real find here: CE004 has been dead since it was written, and CE066 encodes violations that exist today rather than hypothetical ones.
Worth cutting
- The pricing mirror. The four ids it adds to the frontend table were not drift: they were enumerated exemptions in the parity test this PR deletes, which carried its own stale-membership guard. The generator keeps only per-request billing as an exemption axis, so it reproduces three of the seven exclusions and silently overrides the other four. Latent today, since no experiment config references those models. Fix: carry the exemption set into the mirror, or drop the mirror and add the four ids to the frontend table.
durations.py.format_mswas already shared before this PR: both renderers imported it from the formatting module, so there was no duplication to remove. The module's own stated rationale also doesn't hold, since the reports package still pulls the SDK in through models. Fix: leave the function where it was, delete the module and its test.- CE067 and the architecture-notes expansion. Neither is about reports, and the notes additions land in a file the docs PR deletes.
Before merge
- The plugin's framework-contract test.
coder_eval_uipath/eval_runner/tests/skills/test_framework_contract.pyimportorskips the renamed module, so the only guard on the run.json row shape goes green by skipping instead of failing. One-line companion. - Pick a minor bump at dispatch. Four public modules are deleted and
reportsbecomes a package with no aliases, while the release default is patch.
Minor, all optional: reports/helpers.py inherits a sixth of the module this PR dissolves, under a name that invites the same accumulation; the layering rule's relative matcher is depth-blind (latent, nothing nested today); importing the reports package now pulls all five submodules for any one name.
Land this before the docs PR either way: that PR's prose gate rejects two of the modules added here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements
c/2026-09-14-reports-consolidation.md(five phases), thenc/2026-09-14-reports-consolidation-review-fixes.md(four phases) for the residual issues a review of the first found. 15 commits.What changes
Pricing is generated, not copied.
evalboard/lib/pricing.generated.tsis rendered frompricing.pybymake pricing-mirrorand guarded by CE065. Deletes the regex parser, its meta-guard, theDELIBERATELY_UNMIRROREDset, that set's staleness guard, and the comment asking the next reader to keep it honest — five layers of bookkeeping that still shipped a real bug:claude-sonnet-5,gpt-5.6-sol,gpt-5.6-terraandgpt-5.6-lunasat in the exemption set under "the evalboard never runs them" while appearing ~32k / ~2k / ~17k / ~2k times in the run corpus, so every one of those runs rendered—for cost with nothing failing. The one rule the set encoded that was not bookkeeping — that three OpenRouter models must stay unpriced soruns.tsapportions the provider's real bill — becomesModelPricing.per_request_billing, data beside the rate it qualifies.Two verified DRY fixes, and 18 of 19 function-local imports hoisted.
SLOW_PARAMS_PREVIEW_CHARSandTokenUsage.total_tokenseach have one definition again. The 19th import (criteria, which runs pkgutil auto-discovery with registry side effects) stays deferred and is pinned by a subprocess test rather than remembered.reports_stats.pysplits three ways. It was three unrelated modules sharing a file, and the orchestrator importedturn_time_buckets/visible_turn_countfrom it during a run. Now:stats.py(dependency-free by contract — asserted by parsing its AST),result_metrics.py(what the orchestrator reads mid-run), and the report helpers stay. Therun.jsonrow serializer moves torun_record.py; its old home was the only reasonorchestration/batch.pyreached into the reports layer at all.reports/becomes a package (markdown / html / experiment / junit / helpers),format_msmoves todurations.py, and CE066 pins the layering: core may import only the package's public writers. An allowlist, not a denylist. The package publishes 16 names, not 25: nine were only ever imported by sibling modules, which now import each other directly.Documentation truth-up, plus a
## The reports packagesection in.claude/architecture-notes.mdrecording the layering, why each extracted module sits where it does, and the rejected section-data layer with the measurement behind the rejection (only 1 of 4 "duplicated" section pairs shares an input shape).The layering rules now see the whole core layer. CE004 and CE066 share one predicate,
_layers.is_core_path. It was a denylist — ten directory names plus a top-level-module regex — andisolation/was not on it, soisolation/docker_runner.py, thedriver: dockerevaluation path, was invisible to both rules. It is now an anchored allowlist: everything undersrc/coder_eval/is core exceptcli/andreports/, so a new subpackage is core by default. Measured over the real tree: +3 core files (isolation/__init__.py,isolation/docker_runner.py,resources/__init__.py), 0 removals, 0 new violations.TestCoreLayerMembershippins the non-core set against the filesystem.Lint-id bookkeeping is enforced, and CLAUDE.md's tree with it.
[tool.ruff.lint] externalwas missing CE044 and CE065. Both are@pytest.mark.lintclasses, and the existing guard derived rule ids fromALL_RULESalone — blind to exactly that half. It now unions both registries and asserts both directions, so a stale id for a deleted rule fails too.optimize/row, and missingerrors/andplugins.py(the plugin SPI "Adding a New Agent" points at). All three are fixed.Notable
run.jsonrow against a pre-move snapshot.gpt-5.4-mini,gpt-5.4-nano,gpt-5.4-proandgpt-5.5-proto the frontend — those four were pure drift bookkeeping, not a product rule..gitignore's barereports/(meant for run output) also matchedsrc/coder_eval/reports/, leaving__init__.pyuntracked and producing a wheel with zero files undercoder_eval/reports/.node.module, sofrom ..cli import x— this codebase's dominant idiom — fired on nothing, and had since the rule was written. Both rules now share one matcher covering all three spellings: absolute, relative, andfrom coder_eval import cli, which binds the package itself and was missed longest.src/segment (three stopped firing under the anchor, one passed vacuously), and nothing pinned the trailing separator that keeps a module likereports_legacy.pycore — deleting it left all 738 tests acrosstest_custom_lint.pyandtest_lint_runner.pygreen.reports/html.py:825) was the one uncovered line in the renderer. It now has a truncation and a boundary test, both mutation-checked.TestRuffExternalCoversEveryRulenow carries thelintmark like its siblings.make verifyand the Ubuntu lint job still run it; the Windows job (-m "not live and not lint") no longer does..claude/harness-candidates.mdrather than done here, because each changes a rule's scope: CE004 inherits CE066'sreports/exemption through the shared predicate (latent — nothing inreports/importscli), and the anchoredsrc/coder_eval/regex is still duplicated across seven rule modules.Plan errors, corrected by measurement
import coder_eval.reportsdoes not importclaude_agent_sdk" is unachievable:models/agent_config.pyimportsClaudeAgentOptionsand every report module needsmodels. The tests assert what is true instead —durations.pyis SDK-free, andreportsno longer imports the SDK-shapedformattingmodule..stats" contradicted its own Changes body; five is correct (fmt_mean_sdneedsstddev).TestRuffExternalCoversEveryRule, leaving two checks over one list. The existing class was extended instead.Verification
At
7a8951f5:ruff format/checkclean · pyright 0 errors (src + tests configs) · 673 custom lint · 5730 tests at 92.66% coverage · evalboard 803 tests / 54 files / build ✓. At0fabe3fb, before the follow-up commits:make pricing-mirroridempotent on a clean tree · wheel ships all 6coder_eval/reports/files. No follow-up commit touchessrc/,.gitignoreorevalboard/.🤖 Generated with Claude Code