Skip to content

fix(bench): decode XML-escaped paths and names before comparing them - #723

Merged
joyful-ii-V-I merged 2 commits into
mainfrom
lane/bench-719
Nov 22, 2026
Merged

joyful-ii-V-I merged 2 commits into
mainfrom
lane/bench-719

Conversation

@joyful-ii-V-I

@joyful-ii-V-I joyful-ii-V-I commented Nov 22, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #719

Reported by @soarkit, who traced it to bench/arb/run_arb.py: query_files and trace_files took <f p=>, <frame p=> and <hop p=> out with regexes and passed the raw value to norm_path. ripwire's emitter escapes & < > " ' (and tab/newline/CR as numeric references), so src/a&b.py was ranked as src/a&amp;b.py and could never match the decoded gold path. affected_tests already decoded; these two did not.

Change

  • run_arb.py: one xml_path(raw, snap) = norm_path(xml_unescape(raw), snap) used by all three regexes. xml_unescape is the file's existing decoder, the exact inverse of serialize.h escapeXml (five named entities plus numeric references).
  • Sweep of bench/ for the same capture-then-use shape: fixed tgrep-h2h/arms.py (rw_hitset, joined with rg's decoded paths), tokenaudit/sweep.py (derived symbol and file ride back in as CLI args), bench_proof.py (files_in, the path is opened), spec_trace.py (seed symbol). Sites that cannot fail (display-only, dead code, emitter-vs-emitter comparison, identifier-only names) are left; the list is in the lane notes.
  • test/benchxmlpathcheck.sh (registered in regression.sh) drives each fixed reader with a fake ripwire binary emitting <f p="src/a&amp;b.py">, frame and hop forms, and &lt; &gt; &quot; &apos;.
  • changelog.d/719-bench-xml-unescape.md.

Evidence

  • Red before: the gate run against origin/main's bench/ fails 4 arms (src/a&amp;b.py came back undecoded); the fifth arm is the affected_tests control and passes on both. Green after.
  • scripts/qd_range.sh origin/main: PASS, 0 gating rows.
  • Recorded numbers: the ARB dataset, gold and run outputs (bench/external/arb/) are untracked and were not available, so no gold path could be checked and nothing was re-run.

Credit: thanks @soarkit for the report and the exact locations.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Benchmark tools now correctly interpret XML-escaped file paths and symbol names, improving matching when processing encoded benchmark output.
  • Tests
    • Added regression coverage for XML-escaped paths and names across benchmark tools.

…719)

run_arb.py's query_files/trace_files regex-captured <f p=>, <frame p=> and <hop p=> and
passed them to norm_path undecoded, so src/a&b.py ranked as src/a&amp;b.py and never
matched its gold path. One xml_path helper (xml_unescape, the exact inverse of
serialize.h escapeXml, then norm_path) now serves all three; the same capture shape is
fixed in tgrep-h2h rw_hitset, tokenaudit derive_targets, bench_proof files_in and
spec_trace's seed. test/benchxmlpathcheck.sh drives them with a fake binary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 22, 2026 •

Copy link
Copy Markdown

No actionable comments were generated in the recent review.

Walkthrough

Benchmark readers now decode escaped paths and names from ripwire output before using them in rankings, comparisons, and targets. A fake-ripwire test checks the behavior and runs through the regression gate. Dated counts are updated in project documentation and presentation slides.

Changes: Benchmark XML decoding

Layer / File(s) Summary
Decode benchmark paths and names
bench/arb/run_arb.py, bench/bench_proof.py, bench/spec_trace.py, bench/tgrep-h2h/arms.py, bench/tokenaudit/sweep.py
Readers unescape XML-derived paths and names before normalizing paths, comparing results, or using values as targets.
Check decoded values in regression
test/benchxmlpathcheck.sh, test/regression.sh, test/binoverridecheck.sh, changelog.d/719-bench-xml-unescape.md
A fake-ripwire test checks decoded reader results. The regression gate includes the test, and the exemption list records that it uses its own fake ripwire executable.

Snapshot count updates

Layer / File(s) Summary
Update dated counts and census details
README.md, docs/EVALS.md, docs/LIMITS.md, docs/TUNING.md, present/deck5_ripwire_build.js
The documents and presentation update gate-script, cap, and census counts to the 2026-11-22 snapshot.

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix · Severity of issue fixed: Low

Merge Risk: Minimal · up to a5bb0

Benchmark readers now decode XML-escaped paths and names before comparing them, and a regression test covers the change. The remaining edits are documentation count updates. No merge-blocking risk was found.

Pre-merge checks: Passed 3 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Out of Scope Changes check Inconclusive The sibling bench fixes are related. tgrep-h2h/arms.py (rw_hitset), tokenaudit/sweep.py, bench_proof.py (files_in) and spec_trace.py apply the same fix, XML decoding of values captured fro… Confirm the origin of the count snapshot refresh (854 to 865 gate scripts, 440 to 454 caps) against the generator inputs, and show which changes produce it. If the counts reflect unrelated base-branch drift, move the count refresh to a sepa…
Passed Passed checks: 3 passed

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title "fix(bench): decode XML-escaped paths and names before comparing them" matches the main change. The PR adds an xml_path helper that XML-decodes captured values in bench/arb/run_arb.py, and i…
Linked Issues check Passed [#719] Requires XML decoding of the <f p=>, <frame p=> and <hop p=> captures before norm_path, and a test with a path containing &. The PR adds `xml_path(raw, snap) = norm_path(xml_unescape(…
Full details: Title check

Explanation

The title "fix(bench): decode XML-escaped paths and names before comparing them" matches the main change. The PR adds an xml_path helper that XML-decodes captured values in bench/arb/run_arb.py, and it applies the same decode fix to bench_proof.py, spec_trace.py, tgrep-h2h/arms.py, and tokenaudit/sweep.py. The rest of the diff (count snapshot updates, the new benchxmlpathcheck test, and the changelog) supports this fix. The title is specific and concise, and it names the primary change.

Full details: Linked Issues check

Explanation

[#719] Requires XML decoding of the &lt;f p=&gt;, &lt;frame p=&gt; and &lt;hop p=&gt; captures before norm_path, and a test with a path containing &amp;. The PR adds xml_path(raw, snap) = norm_path(xml_unescape(raw), snap) in bench/arb/run_arb.py. query_files uses it for FILE_GROUP_RE. trace_files uses it for FRAME_RE, HOP_RE and FILE_GROUP_RE. xml_unescape is the file's existing decoder, which the PR describes as the inverse of serialize.h escapeXml. It decodes src/a&amp;amp;amp;b.py to src/a&amp;amp;b.py before normalization. The issue suggests html.unescape. The PR reuses the existing decoder, which meets the same requirement. test/benchxmlpathcheck.sh, registered in regression.sh, drives the fixed readers with a fake ripwire binary that emits &amp;amp;amp; paths in frame and hop forms, plus &amp;lt; &amp;gt; &amp;quot; &amp;apos;. Limits: I did not execute the test. The ARB gold data is untracked, so end-to-end ranking against gold is not confirmed. The coding requirement is met by the code and test evidence.

Full details: Out of Scope Changes check

Explanation

The sibling bench fixes are related. tgrep-h2h/arms.py (rw_hitset), tokenaudit/sweep.py, bench_proof.py (files_in) and spec_trace.py apply the same fix, XML decoding of values captured from ripwire output, to the same bug class. The changelog and benchxmlpathcheck gate cover them. The docs and count changes need further evidence. README.md, docs/EVALS.md, docs/LIMITS.md, docs/TUNING.md and present/deck5_ripwire_build.js raise the gate-suite count from 854 to 865. They also raise total caps from 440 to 454 across 149 to 156 files, and undisclosed caps from 152 to 165. The PR adds one test script and changes no src/ file, according to the summary. That does not explain an 11-script increase or a 14-cap increase. docs/counts.tsv is an ignored file, so I did not inspect it. The source of these count changes is therefore not established. Their link to #719 is unverified.

Resolution

Confirm the origin of the count snapshot refresh (854 to 865 gate scripts, 440 to 454 caps) against the generator inputs, and show which changes produce it. If the counts reflect unrelated base-branch drift, move the count refresh to a separate PR or rebase onto current main. Provide the sibling bench-file hunks for direct review if the reviewer wants them verified in this PR.

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 197afc7 and a5bb08e.

Files ignored due to path filters: 1

  • docs/counts.tsv is excluded by !**/*.tsv
Files selected for processing: 6

  • README.md
  • docs/EVALS.md
  • docs/LIMITS.md
  • docs/TUNING.md
  • present/deck5_ripwire_build.js
  • test/binoverridecheck.sh

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

…counts snapshot (#719)

benchxmlpathcheck drives bench Python against a fake script it writes itself and never binds
the ripwire binary, so binoverridecheck's sentinel cannot turn it red: pinned in EXEMPT like
its siblings. docs/counts_snapshot.py refresh clears gendocsmergecheck (D)'s drift bound (11 > 10).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joyful-ii-V-I

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Nov 22, 2026 •

Copy link
Copy Markdown
Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@joyful-ii-V-I

Copy link
Copy Markdown
Collaborator Author

CodeRabbit round on #723 (review on 197afc7): no actionable comments. Commits after the reviewed head:

  • a5bb08e CI only: test/binoverridecheck.sh gets an EXEMPT row for benchxmlpathcheck.sh, which drives a fake ripwire and never binds RIPWIRE_BIN; and docs/counts_snapshot.py refresh, because the gate count had drifted past gendocsmergecheck's bound. No bench reader or source line changes.

Verification at a5bb08e: CI 37 of 37 pass; test/benchxmlpathcheck.sh red on main's bench/ (4 of 5 arms) and green here; binoverridecheck reports 0 false-greens over 814 non-exempt gates.

@joyful-ii-V-I
joyful-ii-V-I merged commit 785143c into main Nov 22, 2026
41 checks passed
joyful-ii-V-I added a commit that referenced this pull request Nov 22, 2026
main moved again before the final build: #723 (#719, bench XML-escaped path/name decode,
its new gate test/benchxmlpathcheck.sh and binoverridecheck exemption). No source or parser
change, so kParserVer stays 342.

Conflicts: README.md, docs/EVALS.md, docs/LIMITS.md, docs/TUNING.md, docs/counts.tsv,
present/deck5_ripwire_build.js — all the generated counts-snapshot regions both sides
refreshed. Kept the train's side, then `python3 docs/counts_snapshot.py refresh` on the
merged tree (tool-written), never text-merged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

XML-encoded file paths stay encoded in --query and --from-trace rankings

1 participant