feat(benchmark): add-enumerate arm — tests whether ADD's single ranked flag is the surfacing ceiling - #191
Open
TinDang97 wants to merge 3 commits into
Open
feat(benchmark): add-enumerate arm — tests whether ADD's single ranked flag is the surfacing ceiling#191TinDang97 wants to merge 3 commits into
TinDang97 wants to merge 3 commits into
Conversation
…d flag is the surfacing ceiling Across three amb1 reps ADD surfaced exactly 1 of 7 planted ambiguities EVERY time — never 0, never 2 — while its PLAN.md template asks for one "Least-sure flag surfaced at freeze", singular and ranked lowest-confidence first. Two readings fit that data: a) the singular flag is a CEILING — ADD noticed more and reported one; b) ADD noticed one. This arm separates them by adding ONE clause to the wrapper: at freeze, list every choice the source spec does not settle, not just the least-sure one. If (a), the rate rises; if (b), it does not and the flag design is exonerated. Either outcome is informative, which is what an A/B needs. Why this and not gate infrastructure: the earlier claim that ADD's disabled human gate was the benchmark's largest distortion does not survive checking. The frozen §3 contract in all three ADD runs contains no trace of the two gaps every build shipped — no ownership, 403, permission or caller-scoping vocabulary in ~3000 characters of contract each. A human at that gate would have approved a document in which both decisions were invisible, because a gate reviews what was written down and a silent gap is by definition not written down. Also checked and NOT acted on: MARKERS contains "clarif", documented as catching spec-kit's NEEDS CLARIFICATION token, while ADD's ⚠ idiom is absent — a real asymmetry in the detector's construction. Adding "⚠" (and an "overlap" anchor) changes NOTHING on these runs: the one sentence it newly matches attributes to A-conflict-response, which ADD already surfaced. A change that would have favoured the method this benchmark belongs to was tested and dropped. Isolation is asserted mechanically, not by eye: the variant minus the clause must equal the baseline byte for byte, so a second smuggled difference fails the suite (verified by mutation). The clause is checked to leak no planted anchor, no domain vocabulary from this workload, and no quota that could be padded. Registered as EXPERIMENTAL: selectable by name, absent from ARM_NAMES, so no default campaign changes cost or composition. Not yet run — a live A/B costs roughly $2 per rep and is the human's call. benchmark/tests: 411 -> 423 passing. author: Tin Dang
…d arm was unrunnable The first launch of add-enumerate failed instantly on `unknown_arm` with 423 tests green. score.py validates arm names against ARM_NAMES independently of the loader and the CLI, and the new suite covered the loader and the CLI only — tested unit, untested seam, the same shape this codebase keeps producing. Cheap this time: the run died before spending anything. It would not have been cheap inside a multi-rep campaign. The guard is now enumerative rather than remembered: no module outside tests may validate against ARM_NAMES, the DEFAULT campaign set. A fourth gate added later fails in the suite instead of at the start of a paid run. author: Tin Dang
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| from benchmark import judge, tamper | ||
| from benchmark.ambiguity import is_implementation_write | ||
| from benchmark.arms.loader import ARM_NAMES | ||
| from benchmark.arms.loader import ALL_ARM_NAMES, ARM_NAMES |
…ntation
add-enumerate's first run reported 3 of 7 surfaced against a baseline of 1 of 7.
Reading the evidence spans, two of the three were junk and the "rise" was
entirely spurious. Three distinct defects, all found by auditing a result that
moved in the direction I wanted:
D1 ANCHORS MATCHED INSIDE LONGER WORDS. "position" is a substring of
"composition", so a sentence about image composition surfaced
A-position-ordering. Anchors are TERMS; they now match on word boundaries.
This also corrupted ATTRIBUTION, which ranks on anchor-hit count — a phantom
hit could outrank and steal a genuine surfacing from the item that earned it.
Markers stay unbounded deliberately: they are stems (assum -> assumption).
D2 A CLOSING XML TAG ACTED AS A MARKER. "assum" is a substring of
"</assumptions>", so the tag ENDING ADD's assumptions block marked whichever
sentence followed it — live, the opening line of the contract body. Simple
tags are now blanked before matching, with EQUAL-LENGTH spaces so every offset
(including edit_pos, which indexes the untouched transcript) still points
where it did. Only well-formed tags: a greedy <...> strip would eat
"start < other.end AND end > other.start", which is prose about a boundary and
frequently the very thing being surfaced.
D3 THE ARTIFACT BUDGET READ SHIPPED DOCUMENTATION. _workspace_artifacts took the
first 40 prose files in sort order. An ADD workspace holds 302, of which 256
are the vendored personas-teacher library — so the budget was consumed
entirely by ADD's own product documentation while PLAN.md, the agent's actual
reasoning, sorted at index 270 and was never read at all. The live false
positive came from design-image-prompt-engineer.md, whose boilerplate reads
"Avoid ambiguous language that could be interpreted multiple ways". That
sentence ships in every ADD workspace; crediting it scores an arm for the
contents of its installer.
Artifacts are now the documents the agent WROTE, per its own tool calls —
arm-neutral by construction, since it asks what the run produced rather than
where a method files things. Fails CLOSED: no transcript means no artifacts,
never "read everything", because reading everything is what caused this.
D3 is the original `artifacts = ()` bug wearing the opposite sign: that one read
none of the agent's documents, this one read everything EXCEPT them. Both hid in
the same place — tests exercised classify() directly and never the seam that
chooses what to feed it.
RE-SCORED, corrected detector:
add (baseline) 0.143 [0.143, 0.143, 0.143] unchanged
spec-kit 0.095 [0.143, 0.143, 0.0] unchanged
add-enumerate 0.143 [0.143] was 0.429
The experiment's apparent lift is gone. Every surviving verdict rests on a
readable sentence that names the contradiction.
Two tests from the earlier artifact fix are AMENDED, not weakened: they now
record the Write in the transcript, while the payload still omits the
recognition — the realistic shape, since an Edit records a slice and the file on
disk ends up saying what no single payload did. Each still passes only by
reading the file.
benchmark/tests: 425 -> 435 passing.
author: Tin Dang
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Across three amb1 reps ADD surfaced exactly 1 of 7 planted ambiguities EVERY
time — never 0, never 2 — while its PLAN.md template asks for one "Least-sure
flag surfaced at freeze", singular and ranked lowest-confidence first. Two
readings fit that data:
a) the singular flag is a CEILING — ADD noticed more and reported one;
b) ADD noticed one.
This arm separates them by adding ONE clause to the wrapper: at freeze, list
every choice the source spec does not settle, not just the least-sure one. If
(a), the rate rises; if (b), it does not and the flag design is exonerated.
Either outcome is informative, which is what an A/B needs.
Why this and not gate infrastructure: the earlier claim that ADD's disabled
human gate was the benchmark's largest distortion does not survive checking.
The frozen §3 contract in all three ADD runs contains no trace of the two gaps
every build shipped — no ownership, 403, permission or caller-scoping vocabulary
in ~3000 characters of contract each. A human at that gate would have approved a
document in which both decisions were invisible, because a gate reviews what was
written down and a silent gap is by definition not written down.
Also checked and NOT acted on: MARKERS contains "clarif", documented as catching
spec-kit's NEEDS CLARIFICATION token, while ADD's ⚠ idiom is absent — a real
asymmetry in the detector's construction. Adding "⚠" (and an "overlap" anchor)
changes NOTHING on these runs: the one sentence it newly matches attributes to
A-conflict-response, which ADD already surfaced. A change that would have
favoured the method this benchmark belongs to was tested and dropped.
Isolation is asserted mechanically, not by eye: the variant minus the clause must
equal the baseline byte for byte, so a second smuggled difference fails the suite
(verified by mutation). The clause is checked to leak no planted anchor, no
domain vocabulary from this workload, and no quota that could be padded.
Registered as EXPERIMENTAL: selectable by name, absent from ARM_NAMES, so no
default campaign changes cost or composition.
Not yet run — a live A/B costs roughly $2 per rep and is the human's call.
benchmark/tests: 411 -> 423 passing.