refactor(reports): generate the pricing mirror and split the reports layer

.claude/commands/coder-eval-code-review-full.md

@@ -348,7 +348,7 @@ in a value that doesn't match the formula.
         - Models with the same field across types (e.g. `RunSummary` and `VariantAggregate`, `TaskDefinition` and `ResolvedTask`, `EvaluationResult` and the per-row `CriterionResult`): verify type, default, validator, and field description match.
         - Parallel orchestration code paths: `orchestration/batch.py` ↔ `orchestration/experiment.py`. A bug fixed in one routinely needs to be fixed in the other (precedent in this codebase: dataset fan-out, run_limits merging, lineage tracking).
         - Parallel agent paths: `Orchestrator` ↔ any new driver (e.g. `isolation/docker_runner.py`) — does the driver preserve the `pending_turn` / `crashed=True TurnRecord` contract documented in CLAUDE.md?
-        - Parallel renderers: `reports.py` ↔ `reports_experiment.py` ↔ `reports_html.py` ↔ `reports_stats.py` — if a new field is added to `EvaluationResult`, do all four render it (and if not, is that deliberate)?
+        - Parallel renderers: `reports/markdown.py` ↔ `reports/experiment.py` ↔ `reports/html.py` ↔ `reports/helpers.py` — if a new field is added to `EvaluationResult`, do all four render it (and if not, is that deliberate)?
       Flag any divergence as a finding even if the unchanged side is technically still correct in isolation — the divergence itself is the bug, and silent drift between parallel paths is one of the most expensive defects to debug later.
 
    3. **Check exhaustiveness when an enum / Literal / status set changes.**
@@ -401,7 +401,7 @@ in a value that doesn't match the formula.
      - **Parallel code paths not updated**: e.g. `orchestration/batch.py` was changed but `orchestration/experiment.py` wasn't, despite handling the same concept; a fix landed in `Orchestrator` but the parallel `DockerRunner` path was missed.
      - **Missing tests for new code paths**: every new branch, public function, validator, CLI flag, and report row needs a test. If a report row was added, is there a test asserting non-zero values for it?
      - **Downstream consumers of changed counting / classification / formula logic**: if a counting formula changed in one place, did all places that compute rates / averages / percentages / pass/fail status from those counts also update? Same for any threshold or scoring change.
-     - **Display / icon / mapping dicts not extended for new enum values**: if a new `FinalStatus` / `AgentState` / `SnapshotMode` / criterion type / category was added, do all rendering dicts in `reports*.py` (`reports.py` / `reports_experiment.py` / `reports_html.py` / `reports_stats.py`) cover it, or do they fall through to `"?"` / `"unknown"`?
+     - **Display / icon / mapping dicts not extended for new enum values**: if a new `FinalStatus` / `AgentState` / `SnapshotMode` / criterion type / category was added, do all rendering dicts in `reports/` (`markdown.py` / `experiment.py` / `html.py` / `helpers.py`) cover it, or do they fall through to `"?"` / `"unknown"`?
      - **Daily/nightly pipeline impact not stated**: if the change touches the production run path (the cron/nightly entrypoint, the `DockerRunner` entrypoint, the `--backend bedrock` judge) or the cross-repo contract consumed by the external `coder-eval-uipath` / eval-runner pipeline (run-record / `task.json` schema, report JSON shape, CLI output), does the PR say what happens to the nightly run? An unstated blast radius on the daily pipeline is itself the gap.
 
      This pass is allowed to surface items that are not tied to a single file:line (since the whole point is that the *absence* of a change isn't anchored anywhere). Express each as a short bullet, prefixed with the bucket it falls into, and reference the *changed* file that triggered the expectation. Each bullet should also carry a severity tag (🔴 / 🟠 / 🟡 / 🔵) using the same anchor table — a missing test for a new public function is 🟠 Test Health; a missing entry in a display dict is typically 🟡; a missing parallel-path update that introduces a real divergence is 🟠 Architecture. **Add these severity tags to the per-axis totals** so they show up in Counts and on-screen output. If there's nothing missing, write a single line: "Nothing identified."

.claude/harness-candidates.md

@@ -450,7 +450,7 @@ with the two `action.yml` items above — one considered change to the action's
   `verify-published-action.yml` reads `task_results[*].status` / `weighted_score` /
   `total_tokens`, and `action.yml`'s score gate reads `weighted_score` / `task_id`.
   These are string keys in shell/YAML that no test or type-checker binds to
-  `eval_result_to_task_dict` (`reports_experiment.py`), so renaming a key there
+  `eval_result_to_task_dict` (`run_record.py`), so renaming a key there
   silently turns an external gate into a no-op — a reviewer here proposed
   `final_status`, which does not exist in `run.json` and would have made a new
   assertion dead on arrival. Guard: assert the key set that non-Python consumers
@@ -859,7 +859,7 @@ re-derive from scratch.
   an existing form. Caught in: the turn-timing consolidation, Phase 5 review.
 
 - [ ] **Pre-existing, surfaced by the turn-timing final review:
-  `reports_stats.regularized_incomplete_beta` clamps an out-of-domain `x`
+  `stats.regularized_incomplete_beta` clamps an out-of-domain `x`
   instead of raising.** Its docstring says "Raises ValueError outside that
   domain — returning NaN would let a bad input render as a real-looking
   statistic downstream", and it does raise for a non-finite `a`/`b`/`x` and for
@@ -916,3 +916,49 @@ re-derive from scratch.
   (AST-comparing every `ClassDef` first line against a base ref) was written
   and used throughout Phase 6 and is the thing to promote if that access
   appears. Caught in: prose mass reduction, Phase 6.
+
+- [ ] A generated surface (`*.generated.*`) has no mechanical guard against being hand-edited
+  — CE065/CE033/CE028 all catch *drift* (source changed, output not regenerated) but an edit
+  to BOTH passes cleanly. Guarding it needs a checksum or a git-attribute gate, not a diff,
+  so it is a different shape of sensor. — caught during the reports consolidation (CE065).
+- [ ] No rule resolves file paths named in PROSE (comments, docstrings, Markdown) across
+  `src/`, `evalboard/`, `litellm/` and `.github/`. That consolidation hand-fixed ~25 stale
+  module references across five phases, and two reviewers each found more the greps missed.
+  The plan's Open Questions measured and declined the CLAUDE.md-only variant (its stale refs
+  live in an ASCII tree, not backticks); a wider variant has the same parsing problem plus
+  legitimate non-resolving refs (container paths, plugin-relative paths). Recorded because
+  the recurrence is now the argument, not the idea. — caught during the reports consolidation.
+- [ ] The anchored package regex `(?:^|[/\\])src[/\\]coder_eval[/\\]` is compiled
+  independently across the rule tree — `ce050_no_union_getattr_probe.py:101`,
+  `ce051_no_driver_override.py:60`, `ce052_process_lethal_must_be_container_gated.py:78`,
+  `ce053_run_record_filename_literal.py:65`, `ce054_env_info_key_round_trip.py:66`,
+  `ce056_no_container_env_literal.py:51` and `ce058_no_timing_literal.py:116` — seven
+  rule modules, to which `_layers.py` adds one more (its `_CLI` and `_REPORTS` derive from it), with the
+  `agents/`-suffixed variant of the same idiom in
+  `_model_ctor.py:28` and `ce059_generation_window_is_two_reads.py:45`, plus a near-variant
+  in `ce037_no_dead_private_helper.py:61` and a `cli/`-suffixed one in
+  `ce048_no_in_process_typer_command_call.py:68`. `_layers.py` is the designated shared rule-helper
+  module, though `_model_ctor.py` is an equal peer and a generic src-path regex arguably
+  belongs in a neutrally named helper rather than one named `_layers`. Not hoisted here
+  because retargeting seven unrelated rules needs a per-rule verification that its scope
+  did not shift — a second refactor inside a review-fix plan. The new copies were written
+  in the established *spelling* deliberately: the defect being fixed was a regex that
+  disagreed with its siblings, so a new variant would be that defect again. — caught during the reports-consolidation review fixes, Phase 1.
+- [x] ~~**CE004 inherits CE066's `reports/` exemption because the two rules share one
+  predicate.**~~ **DONE.** `_layers.py` now shares the package anchor and the `cli/`
+  boundary (`is_package_path`, `is_cli_path`) rather than one exemption set. CE066 keeps
+  `is_core_path` (`{cli, reports}` exempt); CE004's scope is the package minus `cli/`.
+  Widening CE004 to `reports/` found 0 violations. `test_the_reports_package_is_in_scope`
+  and `TestCoreLayerMembership.test_ce004_scope_is_every_module_outside_cli` both fail if
+  CE004 goes back to the core predicate; `test_the_reports_package_itself_stays_exempt`
+  pins that CE066's scope did not widen with it. — caught in the reports-consolidation
+  review fixes, Phase 1 quality review.
+
+- [ ] **A CLAUDE.md Directory Structure bullet naming a path that no longer exists.**
+  Shipped briefly as CE067 over the fenced `coder_eval/` tree, then removed when that
+  tree was replaced by `ls` plus selective bullets — the exhaustive half of the rule
+  became false by design. The surviving half is still real: the bullets name modules
+  (`result_metrics.py`, `reports/html.py`, `models/container_paths.py`) and a rename
+  leaves them stale with nothing failing. Needs a backtick-path extractor scoped to
+  one section, which is the narrow case of the prose-path candidate above. — caught
+  during the reports consolidation rebase.

.claude/notes/orchestration.md

@@ -48,7 +48,7 @@
   `run` and `execute` share one body (`run_command.run_pipeline`) and differ solely in
   that flag, so there is no third code path. Three things are refused rather than
   degraded: `--junit-xml` (a report of verdicts, and there are none — though
-  `reports_junit` still emits `<skipped>` for an ungraded row it encounters),
+  `reports/junit.py` still emits `<skipped>` for an ungraded row it encounters),
   `--allow-host-grading` (it decides how an ungraded row is GRADED, and `execute` grades
   nothing), and simulation tasks (their turn-continuation logic reads criteria results,
   so an ungraded dialog would silently change its own stopping behavior). `stop_early:`

.claude/notes/reporting.md

@@ -184,11 +184,80 @@ malformed record — which raises while building the per-call breakdown — abor
 with the run untouched, matching the caller's "keeping static pricing" contract. Spend
 tagged with an iteration no turn has is surfaced rather than silently dropped.
 
+## The reports package
+
+`reports/` is a **leaf**: it may import from anywhere in `coder_eval`, and the core layers
+may import only its public *writer* entry points. That asymmetry is the whole point, and
+**CE066** enforces it. The invariant is not "core must not import reports" — core
+legitimately *writes* reports (`orchestrator.py` writes the per-task HTML,
+`orchestration/batch.py` drives `ReportGenerator`). It is that a **metric, a statistic, a
+serializer or a formatter** must never be reached out of the rendering layer.
+
+That was the actual shape of the code before the split. `reports_stats.py` was three
+unrelated modules sharing a file, and the orchestrator imported `turn_time_buckets` and
+`visible_turn_count` from it *during a run* — a number the evaluation loop needs, living in
+a reporting module. The three pieces now sit where their consumers are:
+
+- **`stats.py`** — distribution-free statistics, **dependency-free by contract**: stdlib
+  only, no `coder_eval` import, direct or relative. A unit test parses its AST and asserts
+  that, rather than leaving it to convention, because being reasonable-about-in-isolation is
+  the only reason it is a separate module. Display formatters (`fmt_mean_sd`, `fmt_p`) stay
+  in `reports/helpers.py`: they return `"N/A"`, `"—"` and `"<0.001"`, which is presentation.
+- **`result_metrics.py`** — metrics derived from a finished `EvaluationResult`, consumed by
+  the orchestrator mid-run as well as by the reporters. Deliberately **not** folded into
+  `timing.py`, which has no `EvaluationResult` dependency and is imported by every agent
+  adapter; adding one would widen that surface for everyone.
+- **`run_record.py`** — the `run.json` task-row serializer. It is a run-record serializer,
+  not a report, and its old home inside the experiment reporter was the *only* reason
+  `orchestration/batch.py` reached into the reports layer at all. Moving it is what lets
+  CE066's allowlist be purely writers; carrying a serializer on that list would be the rule
+  documenting a wart instead of the wart being removed.
+
+**CE066 checks both the absolute and the relative import spelling.** Its first draft matched
+only `node.module`, which for `from ..reports import X` holds `"reports"` with the dots in
+`node.level` — so it fired on neither of the two real edges in the tree, and its own tests
+passed because they used the absolute form. The layer predicates live in
+`tests/lint/rules/_layers.py` so CE004 and CE066 cannot drift about where the package or its
+`cli/` boundary is. Each rule's scope is an allowlist of what is *exempt*, so a new subpackage
+is in scope by default: CE066's core is *everything under `src/coder_eval/` except `cli/` and
+`reports/`*, and CE004's scope is *everything except `cli/`*. The two sets differ on purpose —
+the reports package runs without the CLI, so it must not import `cli`, but it may reach into
+itself. CE004 first borrowed CE066's predicate whole and so inherited the `reports/`
+exemption. Both denylist forms before that also leaked: naming only `orchestrator.py` left
+`result_metrics.py` exempt (the module CE066's own fix message points at), and its
+ten-directory successor never named `isolation/`, leaving the `driver: docker` evaluation path
+invisible to both rules.
+
+**`format_ms` lives in `durations.py`, not `formatting.py`.** `formatting.py` imports
+`claude_agent_sdk` for the payload formatters, and the reports package should not reach
+through an SDK-shaped module for a 14-line duration formatter. This does *not* make the
+package SDK-free — `models/agent_config.py` imports `ClaudeAgentOptions` and every report
+module needs `models` — so the tests assert what is true: `durations.py` is SDK-free, and
+`reports` no longer imports `coder_eval.formatting`.
+
+### Rejected: a shared section-data layer
+
+The markdown and HTML reporters render four "duplicated" sections. All four pairs were read
+in full before deciding, and **only one shares an input shape** (command statistics, both
+taking `CommandStatistics`); the others take a `list[dict]` row, a `TokenUsage`, an
+`EvaluationResult` and a `list[EvaluationResult]` across three different scopes. The
+remaining differences are legitimate per-surface presentation, not drift: `:.1f%` vs `:.0f%`,
+and an unmeasured average **hidden** in markdown versus **dashed** in HTML — two valid
+renderings of the same `None`. Building the adapter would mean normalizing dict-row and
+live-model inputs across three scopes, touching the `run.json` contract, to remove about
+twenty lines. Rejected on KISS/YAGNI. The two things in those pairs that *were* real — a
+literal `50` beside its own `SLOW_PARAMS_PREVIEW_CHARS`, and a hand-rolled
+`TokenUsage.total_tokens` — were simply fixed.
+
+`analysis.py` and `formatting.py` stay top-level on purpose: they are not report modules,
+and moving them in would give the package an SDK dependency and force CE066 to exempt the
+orchestrator's `analysis` import.
+
 ## Report rollups and the HTML twin
 
-`reports_html.py` is the evalboard's STATIC TWIN: the two render the same run and must
+`reports/html.py` is the evalboard's STATIC TWIN: the two render the same run and must
 agree, so a rule implemented on one side belongs on the other. The arithmetic itself lives
-in `reports_stats.py` and the renderers only format it.
+in `result_metrics.py` and `stats.py`; the renderers only format it.
 
 ### An unmeasured value is never zero
 

.claude/notes/timing.md

@@ -144,7 +144,7 @@ per-harness composition.
 
 The span set the generation subtraction, the head and the tail are all measured against,
 so they cannot disagree about which calls exist. Shared with
-`reports_stats.turn_time_buckets`, which answers the same question about a finished
+`result_metrics.turn_time_buckets`, which answers the same question about a finished
 `TurnRecord` — a second typed copy of this rule is how two report surfaces come to publish
 two different tool totals for one run. (`scripts/timing/decompose_run.py` keeps its own,
 over raw `task.json` dicts rather than models; that is the sanctioned third reader, and

.github/workflows/pr-checks.yml

@@ -181,19 +181,21 @@ jobs:
           echo "📊 All checks passed: formatting, linting, types, security, tests"
 
   evalboard:
-    # The dashboard's own gate. `evalboard/` ships ~460 vitest assertions,
-    # including the pricing drift guard that asserts lib/pricing.ts still agrees
-    # with src/coder_eval/pricing.py — and until this job existed NOTHING ran
-    # them: not a workflow, not a Makefile target, not a pre-commit hook. The
-    # guard was consequently red on `main` for weeks while a 3x-wrong Opus rate
-    # and five unpriced in-use models shipped to the board. An unrun assertion is
-    # documentation, not enforcement.
+    # The dashboard's own gate. `evalboard/` ships ~460 vitest assertions, and
+    # until this job existed NOTHING ran them: not a workflow, not a Makefile
+    # target, not a pre-commit hook. The pricing guard was consequently red on
+    # `main` for weeks while a 3x-wrong Opus rate and five unpriced in-use models
+    # shipped to the board. An unrun assertion is documentation, not enforcement.
+    #
+    # Rate-table drift is no longer this job's concern: lib/pricing.generated.ts
+    # is GENERATED from src/coder_eval/pricing.py, and CE065 in `quality-gate`
+    # fails a reprice that was not regenerated. What runs here is the CONSUMPTION
+    # half (pricing-generated.test.ts) — a generated file that is missing, empty
+    # or narrow fails the board's own build.
     #
     # Deliberately NOT path-filtered. `paths:` is workflow-scoped in GitHub
-    # Actions, and the parity guard's whole point is that a reprice in
-    # src/coder_eval/pricing.py — a pure-Python diff touching no evalboard file —
-    # must trip it. A `evalboard/**`-only filter would skip exactly the change
-    # class this job exists to catch.
+    # Actions, and a skipped required check blocks a PR rather than passing it —
+    # so the filter buys nothing and costs a merge-blocking pending status.
     name: Evalboard (Types, Tests, Build)
     # Fork-PR carve-out — see `quality-gate`. `pnpm install --frozen-lockfile` runs
     # the PR's own lockfile install scripts, same untrusted-code class.

.github/workflows/verify-published-action.yml

@@ -463,7 +463,7 @@ jobs:
           # The JUnit report must describe the same run, not merely be parseable: an empty
           # but well-formed <testsuites/> passed the old parse-only check. Trusted,
           # self-generated input (our writer emits no DTDs/entities), so stdlib ET is fine.
-          # `>=` not `==`: reports_junit.py also emits synthetic `skipped` / `suite-gates`
+          # `>=` not `==`: reports/junit.py also emits synthetic `skipped` / `suite-gates`
           # testsuites, which only ever ADD cases.
           cases = len(list(ET.parse(os.environ["JUNIT"]).getroot().iter("testcase")))
           if cases < len(rows):

.gitignore

@@ -61,7 +61,9 @@ uipath.json
 .claude/settings.local.json
 c/
 runs/
-reports/
+# Anchored to the repo root: this is generated RUN OUTPUT, not the
+# `src/coder_eval/reports/` package, which a bare `reports/` also matched.
+/reports/
 /runs/**/artifacts/
 .DS_Store
 tmp/