feat: add benchmark/analyze_top.py, the memory-vs-latest/mitigation analysis script - #67
Draft
algomaster99 wants to merge 2 commits into
Draft
algomaster99 wants to merge 2 commits into
algomaster99 wants to merge 2 commits into
Conversation
algomaster99
changed the title
algomaster99
force-pushed
the
feat/maven-property-pin-resolution
branch
from
September 25, 2026 22:11
fc55d31 to
c9dd91d
Compare
algomaster99
marked this pull request as draft
September 25, 2026 22:15
algomaster99
force-pushed
the
feat/maven-property-pin-resolution
branch
from
September 26, 2026 16:13
c9dd91d to
c8aafaf
Compare
Walks a run_all.sh output dir and reports, per rep's final manifest, whether
the target dependency ended up satisfied (an exact pin at latest, or a range
that already covers latest - yul's own pins.Diff checks both) and how it got
there: the model already knew the version (native), it shelled out to a
registry/package-manager CLI to look it up (tool), or yul's hook blocked the
write and forced the correction (hook, hook condition only).
Reuses pkg/maven/pom.go's ${property} and <parent> POM resolution logic so
Maven pins are classified the same way yul's own checker does, and treats a
GitHub Actions commit-SHA pin as exact (it's yul's own preferred correction,
not an unpinned reference).
For the hook condition, also reports reps that never ended up satisfied,
split into a genuine miss (pin present but still outdated - none observed)
vs. the target package never appearing in the final manifest at all, in
which case it searches the manifest for the package's short name elsewhere
as a signal of a swap (e.g. junit:junit -> org.junit.jupiter:junit-jupiter)
rather than a silent drop.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
algomaster99
force-pushed
the
feat/maven-property-pin-resolution
branch
from
September 26, 2026 18:10
c8aafaf to
eaaa01f
Compare
…sion parsing in analyze_top.py - classify reps that solved the task with an equivalent alternative package/mechanism (e.g. junit-jupiter instead of junit, tzdata instead of pytz, cache: npm instead of actions/cache) as satisfied via a new 'alternative' route, instead of counting them as a hook/tool miss - exclude reps where the model never declared any dependency at all (it implemented the functionality itself) from the denominator, since there's no pin for yul to have acted on - fix find_pin_pypi_pyproject to handle multi-constraint PEP 621 specs like "click>=8.1,<9" instead of misreporting the package as absent Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
benchmark/analyze_top.py, which walksbenchmark/top-final(or anyrun_all.shoutput dir) and, per rep dir, classifies each target dependency's final manifest pin (exact/range/parent-POM/none) across all six ecosystems, compares it against the package's latest registry version, and cross-references transcripts for mitigation events, bash-write attempts, curl/wget use, package-manager-CLI use, and cross-run "cheating" signals — producing the memory-vs-latest / hook-mitigation table used in the paper.Includes Maven
${property}version-pin resolution:find_pin_mavenpreviously treated a<version>${x.version}</version>reference as an unresolved "range", even when the property itself resolves to a single exact version in the POM's own<properties>block — so the "versioned" metric silently undercounted every Maven case using this idiomatic pattern (lombok,slf4j,kotlin-stdlib-jdk7).Ports
pkg/maven/pom.go'smavenPinnedVersion/parsePOMPropertiesresolution logic into the analysis script so it classifies these pins the same way yul's own checker does. Also applies the same resolution tofind_parent_maven(parent-POM pin tracking) for consistency.Before/after on
benchmark/top-final, per-ecosystemmavenrow:Not addressed here: the remaining
junit/mysql-connector-javareps still showpin=None, but that's a separate issue — the case metadata targets legacy Maven coordinates (junit:junit,mysql:mysql-connector-java) while Claude correctly writes the modern relocated coordinates (org.junit.jupiter:junit-jupiter,com.mysql:mysql-connector-j), sofind_pin_mavennever finds a matching<dependency>block at all.🤖 Generated with Claude Code