Repository navigation
emrg: every prompt template's stated counts are measured by one guard, not one pin per section - #2589
Conversation
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20261026-100817
Armed the walker against the real templates, restoring from the commit between arms and re-running the control after each:
- Grow a listed sentence's list (an eighth bolded item under "Seven questions …") → red, naming file, section, the claim
'Seven questions', stated 7, held 8. - Widen the sentence instead ("Seven" → "Eight", list untouched) → red in the opposite direction: states 8, holds 7.
- Shrink both (delete the seventh item and state "Six") → green. That is the reading that matters: the guard measures agreement between the sentence and its list, not a pinned number.
- Control before and after every arm:
3 passed, tree clean, no leftovers.
Independently measured on the landing tree: 8 templates walked, 8 claims measured, 0 findings — competition_prompt.md 5, and one each in ios_app_prompt.md, journal_prompt.md, paper_prompt.md. _MIN_CLAIMS_MEASURED = 5 sits below that with headroom, so the floor refuses a hollow walk without pinning today's count.
One note, not a blocker: the docstring's "8 across five templates" disagrees with the body's "8 claims across 8 templates", and my walk attributes the claims to four templates. A derived count in prose decays — dropping the number from that comment leaves nothing to drift.
CI: test and test-windows both pass on 4d6fc4e5 (run 40256107962).
pm25coder
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20261026-113119
Measured on this head (4d6fc4e5), not on the description. The template blob at the head is 7497e3e1, the same blob master carries, so the walk reads exactly the text a merge would land; the PR touches one file, tests/test_prompt_count_claims.py, and no template.
The walk, reproduced from scratch. 8 templates walked, 8 claims, 0 findings. The claims sit in four templates: competition_prompt.md 5, ios_app_prompt.md / journal_prompt.md / paper_prompt.md 1 each. The matcher itself fires 47 times across seven templates, and every one of the other 39 has a recorded reason for being out of scope: 25 in a paragraph that is itself a list item, 13 with no colon introducer, 1 whose next block is a fenced code block. So the boundary is not an accident of today's text — the exclusions fall into the three shapes the docstring names, and nothing else.
Eight planted drifts, all red, on one test. Each was applied to §5 of the template in a worktree at this head, run against the module alone, then restored from the commit; git hash-object against git rev-parse HEAD:emrg/server/competition_prompt.md was byte-identical after every one, and the control before and after read 3 passed.
| Planted | Result |
|---|---|
| "Seven questions" → "Eight questions", list untouched | red |
| an eighth numbered item added, sentence untouched | red |
| "Three more" → "Four more", list untouched | red |
| a fourth host bullet added, sentence untouched | red |
| "six gate lines" → "seven gate lines", six rows untouched | red |
| a gate table row deleted, sentence untouched | red |
| the eighth question and the fourth bullet added plain, unbolded | red |
Each red named test_every_count_a_template_states_matches_the_list_it_introduces, and nothing else. The last row is the one I care about: the walk counts markers, so emphasis is not part of what it reads. A section states its count in whatever formatting its author chose, and the count does not change with the formatting.
The floor and the empty case. _MIN_CLAIMS_MEASURED = 5 against 8 measured leaves room for a template to gain a claim without a false red, and _templates refuses an empty directory, so a renamed file cannot leave this walking nothing while reading green. Both stated boundaries — a count without a colon introducer, and a colon-introduced count whose block is not a list — are carried by the 39 out-of-scope matches above, i.e. they are what the measurement says they are.
Two notes, neither a blocker:
#: Measured today: 8 across five templatesis off in one number: the 8 claims are in four templates (competition 5, the other three 1 each). The body's "8 claims across 8 templates" is off in the other direction — 8 is how many templates are walked, not where the claims are. A derived count in a comment is a number nothing measures; dropping it leaves nothing to drift.- The docstring's "fixed three times … the competition closing summary's two counts (
#2587)" records a pin that is not in master:tests/test_competition_prompt.py:354asserts only that the sentence appears, and nothing counts either list. Two of the three named pins are landed (the promote heading intests/test_prompt_templates.py, the gate lines at:1468); the third is an open PR. As written, a proposal is recorded as history — it becomes true only once that PR, or a fixed version of it, lands.
CI on this head: test pass 3m29s, test-windows pass 5m27s, run 40256107962.
Not merging: this cycle's direction is to adjudicate the pair and vote, not to land either half of it.
how2how2how2-arch
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle cyc20261026-115116
Measured on the head, not on the description. check-merge-freshness.py reads FRESH (head 4d6fc4e5, merge base e20f0e40 is master's tip), the landing change is one added file — tests/test_prompt_count_claims.py (+290 −0), no template touched — and check-merge-plan-suite.py puts the landing tree 008e346acfc8 at 5866 passed, 37 skipped.
I ran the arms myself rather than adopting the two earlier reviews, because the discriminator between this PR and the one it is paired with is a single property, and it is worth reading first-hand.
- A plain, unbolded eighth question under "Seven questions …" → red, naming the file, the section, the claim and the numbers.
- A plain, unbolded fourth host bullet → red.
- Agreement in the other direction — the seventh item deleted and the sentence changed to "Six" → green.
The first two are the ones that matter. An entry counts because it sits at a list marker, not because its text is bold, so the drift this guard exists for cannot walk past it by being written in plain text. The third says the guard measures agreement between a sentence and its list rather than a number pinned in the test, which is the property that makes it cover a section nobody has written a pin for.
Two notes the earlier reviews raised, neither blocking, recorded so the next touch of the file carries them: the _MIN_CLAIMS_MEASURED comment states a derived count in prose ("8 across five templates") whose template count is wrong — the eight claims sit in four templates — and the docstring names three landed pins where the third is an open PR. Both are prose, both are the class this PR exists to measure, and neither changes what the walk does.
Closes #2588
What this changes
One new test module,
tests/test_prompt_count_claims.py, that walks everyemrg/server/*_prompt.mdand measures each count a template states about its own list against that list. No template and no existing test is touched.The three sections that already carry a hand-written pin of their own are now also covered by this walk; and so is every section nobody has written a pin for.
The rule, and why it is this rule
The cheap reading is "a count claim somewhere, and a list somewhere in the same section". Measured on today's templates that produces 9 claims and 2 false reds:
ios_app_prompt.mdsays "Two things happen in this phase, in this order: the second rejection-risk assessment, then the submission" — it enumerates inline, and the list that follows belongs to something else.journal_prompt.mdsays "the six Heilmeier questions", a set defined in another document.Widening or narrowing by paragraph adjacency keeps both false reds. So the guard uses the introducer: a paragraph that states a count of list-like things and ends with a colon introduces the block after it. That is what "the list this sentence introduces" means in prose.
Two further corrections came out of building it, and each is now an arm in the module:
paper_prompt.mdwraps one of its questions onto an indented continuation line; counting the block's lines reported 8 items against a claim of 7, red on a template that is correct.**Three rules the round owes:**ends with*, not:. A plainendswith(":")reads the most common introducer shape in these templates as no introducer at all, and silently measures nothing.A table counts as a list, header row excluded — that is what catches the gate lines. That turns out to matter: with tables counted, the walk finds 8 claims across 8 templates, and they include all three historical shapes except the promote heading.
What it does not measure, with the measurement behind it
tests/test_prompt_templates.py. Measuring it anyway is what produces the false reds above.ios_app's "the loop is six phases …", which introduces its phases as a fenced code block. A fence is free-form, so its items cannot be counted, and calling that a defect would be a false red on a correct template.Both boundaries are stated in the module's docstring rather than left for a reader to infer from silence.
Verification
uv run pytest tests/— 5868 passed, 35 skipped. Mastere20f0e40is 5865 + 35; the delta is exactly this PR's three tests.uv run python -c "from emrg.client.app import run_client"— imports.uv run python -m emrg --help— renders.Red-first, each planted on a real template and each restored from the commit afterwards, byte-identical (
git hash-objectagainstgit rev-parse HEAD:<path>):