Skip to content

fix(scanner): enforce the verdict schema where the rows are written - #157

Merged
elfrost merged 1 commit into
mainfrom
fix/verdict-corpus-enforcement
Sep 22, 2026
Merged

elfrost merged 1 commit into
mainfrom
fix/verdict-corpus-enforcement

Conversation

@elfrost

@elfrost elfrost commented Sep 22, 2026

Copy link
Copy Markdown
Owner

Follow-up to #154 (ADR-016), driven by its first production run.

What the first run showed

Scan #109 (overwirehq/claude-code-telegram, 2026-09-22) produced verdicts.json with 7 scanner rows and 6 curation rows — the plumbing worked end to end, and the published post carried both new blocks.

Every one of the 6 curation rows was invalid:

Defect Rows
rule empty 6/6
verdict free prose ("by-design not-reachable", "FP - parameterized") 6/6
confirmed-real used for 13 dependency CVEs under a "0 real" headline 1

The test suite was green throughout. It exercises the constructor; production does not — the curation half is hand-written JSON, so VerdictRecord.__post_init__ never runs. A closed vocabulary is only closed if something closes it.

A fourth problem is structural rather than drift: reports/ is gitignored, so the corpus lived on one machine and would not survive a clone. ADR-014's measurement only worked because the archived reports happened to still be on disk.

Fixes

  • scanner/verdict_corpus.py + scanner/run_verdict_check.py — --check reports every problem in every row in one pass and exits 2. /daily Phase 4 must see exit 0 before publishing. A checker that reveals one defect per run costs as many runs as there are defects, so fields are checked independently rather than through the first-raising constructor.
  • rule required on every record. ADR-014 needed sqlalchemy-execute-raw-query fired 157 times, not "some rules were dismissed".
  • dependency-currency added; confirmed-real documented as first-party only.
  • corpus/verdicts/ committed — the durable half. reports/ stays ignored (it holds raw dumps and unsent disclosure drafts); a negation rule there would need a fragile reports/* + !reports/*/ ladder that risks exposing reports/disclosures/. --summary counts the corpus.

scanner/verdicts.py hit 296 of the 300-line ceiling, so validation and corpus loading moved out.

Not fixed, on purpose

Scan #109's six invalid rows are left as they are. Repairing them would mean inventing rule identities that were never recorded, and a corpus with one fabricated entry is worth less than one with a known hole. Scan #110 is the first valid entry.

Verification

🤖 Generated with Claude Code

ADR-016 shipped the dismissal corpus on 2026-09-21. Its first production run,
scan #109 on 2026-09-22, produced verdicts.json with 7 scanner rows and 6
curation rows — the plumbing worked — and every curation row was invalid. All
six had an empty `rule`; all six carried free prose in `verdict`
("by-design not-reachable", "FP - parameterized", "real - lockfile refresh").

The unit tests passed the whole time, because they exercise the constructor and
production does not: the curation half is hand-written JSON, so
VerdictRecord.__post_init__ never runs. A closed vocabulary is only closed if
something closes it.

Four fixes:

- `scanner/verdict_corpus.py` + `scanner/run_verdict_check.py`. `--check`
  reports every problem in every row in one pass and exits 2; `/daily` Phase 4
  must see exit 0 before publishing. A checker that reveals one defect per run
  costs as many runs as there are defects.
- `rule` is now required on every record — ADR-014 needed
  "sqlalchemy-execute-raw-query fired 157 times", not "some rules were
  dismissed".
- `dependency-currency` joins the vocabulary; `confirmed-real` is documented as
  first-party only. Scan #109 filed 13 dependency CVEs as `confirmed-real`
  under a headline of zero real findings.
- `corpus/verdicts/` is committed. `reports/` is gitignored — it holds raw
  dumps and unsent disclosure drafts — so the corpus lived on one machine and
  would not survive a clone. `--summary` counts it.

`scanner/verdicts.py` hit 296 of the 300-line ceiling, so validation and corpus
loading moved to the new module.

Scan #109's six invalid rows are left alone: rewriting them would mean
inventing rule identities that were never recorded, and a corpus with one
fabricated entry is worth less than one with a known hole.

ADR-017. 26 new tests; 416 pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@elfrost
elfrost merged commit 92f16ec into main Sep 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant