Skip to content

chore: add top-final benchmark run results - #68

Open
algomaster99 wants to merge 1 commit into
feat/maven-property-pin-resolutionfrom
bench/top-final-results
Open

algomaster99 wants to merge 1 commit into
feat/maven-property-pin-resolutionfrom
bench/top-final-results

Conversation

@algomaster99

@algomaster99 algomaster99 commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

Adds the full top-package benchmark run (60 cases × hook/nohook × 3 reps) under benchmark/top-final/, and includes the output of python3 benchmark/analyze_top.py against these results:

====================================================================================================
SUMMARY
====================================================================================================
Condition  | Ecosystem      | Satisfied  | Native  | Tool   | Hook   | Alt   | via range
---------- | -------------- | ---------- | ------- | ------ | ------ | ----- | ---------
nohook     | all            | 127/175    | 8       | 106    | 0      | 13    | 36       
nohook     | cargo          | 29/29      | 2       | 25     | 0      | 2     | 21       
nohook     | githubactions  | 3/30       | 0       | 1      | 0      | 2     | 0        
nohook     | go             | 30/30      | 0       | 30     | 0      | 0     | 0        
nohook     | maven          | 19/30      | 5       | 8      | 0      | 6     | 0        
nohook     | npm            | 23/26      | 0       | 22     | 0      | 1     | 10       
nohook     | pypi           | 23/30      | 1       | 20     | 0      | 2     | 5        
hook       | all            | 175/175    | 16      | 75     | 72     | 12    | 19       
hook       | cargo          | 29/29      | 4       | 17     | 5      | 3     | 13       
hook       | githubactions  | 30/30      | 0       | 0      | 27     | 3     | 0        
hook       | go             | 29/29      | 2       | 26     | 1      | 0     | 0        
hook       | maven          | 30/30      | 7       | 3      | 14     | 6     | 0        
hook       | npm            | 27/27      | 2       | 16     | 9      | 0     | 4        
hook       | pypi           | 30/30      | 1       | 13     | 16     | 0     | 2        

Satisfied's denominator excludes 5 reps per condition where the model never declared any dependency at all (it implemented the target functionality itself - there's nothing for yul to have acted on). The new Alt column is the subset of Satisfied reached via a different-but-equivalent package or mechanism than the one the case targeted (e.g. junit-jupiter instead of junit, tzdata instead of pytz, GitHub Actions' cache: npm instead of actions/cache) - a legitimate solution, not a hook/tool miss. Both classifications came out of manual review (see PR discussion below) and are hardcoded in analyze_top.py as EXCLUDED_REPS/ALTERNATIVE_REPS, keyed per (case, condition, rep) since hook and nohook are independent runs.

====================================================================================================
NOHOOK CONDITION: reps that did NOT end up satisfied
====================================================================================================

Pin present but still outdated / range excludes latest: 45 reps
  - ghactions-top-01-checkout/rep1: actions/checkout (exact) = v5
  - ghactions-top-01-checkout/rep2: actions/checkout (exact) = v4
  - ghactions-top-01-checkout/rep3: actions/checkout (exact) = v5
  - ghactions-top-02-setup-node/rep1: actions/setup-node (exact) = v4
  - ghactions-top-02-setup-node/rep2: actions/setup-node (exact) = v4
  - ghactions-top-02-setup-node/rep3: actions/setup-node (exact) = v4
  - ghactions-top-03-upload-artifact/rep1: actions/upload-artifact (exact) = v4
  - ghactions-top-03-upload-artifact/rep3: actions/upload-artifact (exact) = v4
  - ghactions-top-04-setup-python/rep1: actions/setup-python (exact) = v5
  - ghactions-top-04-setup-python/rep2: actions/setup-python (exact) = v5
  - ghactions-top-04-setup-python/rep3: actions/setup-python (exact) = v5
  - ghactions-top-06-setup-java/rep1: actions/setup-java (exact) = v4
  - ghactions-top-06-setup-java/rep2: actions/setup-java (exact) = v4
  - ghactions-top-06-setup-java/rep3: actions/setup-java (exact) = v4
  - ghactions-top-07-docker-login/rep1: docker/login-action (exact) = v3
  - ghactions-top-07-docker-login/rep2: docker/login-action (exact) = v3
  - ghactions-top-07-docker-login/rep3: docker/login-action (exact) = v3
  - ghactions-top-08-download-artifact/rep1: actions/download-artifact (exact) = v4
  - ghactions-top-08-download-artifact/rep2: actions/download-artifact (exact) = v4
  - ghactions-top-08-download-artifact/rep3: actions/download-artifact (exact) = v4
  - ghactions-top-09-docker-buildx/rep1: docker/setup-buildx-action (exact) = v3
  - ghactions-top-09-docker-buildx/rep2: docker/setup-buildx-action (exact) = v3
  - ghactions-top-09-docker-buildx/rep3: docker/setup-buildx-action (exact) = v3
  - ghactions-top-10-docker-build-push/rep1: docker/build-push-action (exact) = v6
  - ghactions-top-10-docker-build-push/rep2: docker/build-push-action (exact) = v6
  - ghactions-top-10-docker-build-push/rep3: docker/build-push-action (exact) = v6
  - maven-top-05-lombok/rep1: org.projectlombok:lombok (exact) = 1.18.38
  - maven-top-05-lombok/rep2: org.projectlombok:lombok (exact) = 1.18.38
  - maven-top-05-lombok/rep3: org.projectlombok:lombok (exact) = 1.18.38
  - maven-top-07-slf4j/rep1: org.slf4j:slf4j-api (exact) = 2.0.17
  - maven-top-07-slf4j/rep2: org.slf4j:slf4j-api (exact) = 2.0.16
  - maven-top-07-slf4j/rep3: org.slf4j:slf4j-api (exact) = 2.0.16
  - maven-top-09-kotlin-stdlib-jdk7/rep1: org.jetbrains.kotlin:kotlin-stdlib-jdk7 (exact) = 1.6.21
  - maven-top-09-kotlin-stdlib-jdk7/rep2: org.jetbrains.kotlin:kotlin-stdlib-jdk7 (exact) = 1.6.21
  - maven-top-09-kotlin-stdlib-jdk7/rep3: org.jetbrains.kotlin:kotlin-stdlib-jdk7 (exact) = 2.4.10
  - maven-top-10-h2database/rep1: com.h2database:h2 (exact) = 2.3.232
  - maven-top-10-h2database/rep3: com.h2database:h2 (exact) = 2.3.232
  - npm-top-10-fresh/rep1: fresh (exact) = 0.5.2
  - npm-top-10-fresh/rep2: fresh (exact) = 0.5.2
  - npm-top-10-fresh/rep3: fresh (exact) = 0.5.2
  - pypi-top-01-requests/rep2: requests (exact) = 2.32.3
  - pypi-top-01-requests/rep3: requests (exact) = 2.32.3
  - pypi-top-03-numpy/rep1: numpy (exact) = 2.4.6
  - pypi-top-03-numpy/rep3: numpy (exact) = 2.4.6
  - pypi-top-10-pandas/rep1: pandas (exact) = 2.2.3

Target package not found at all in the final manifest: 3 reps
  - ghactions-top-05-cache/rep3: actions/cache absent, no trace of it anywhere (likely never added)
  - pypi-top-02-six/rep1: six absent, no trace of it anywhere (likely never added)
  - pypi-top-02-six/rep3: six absent, no trace of it anywhere (likely never added)

Counted as satisfied via an alternative (equivalent) package/mechanism, per manual review: 13 reps
  - cargo-top-09-winapi-x86_64-pc-windows-gnu/rep1: winapi-x86_64-pc-windows-gnu absent, but an equivalent alternative was used instead
  - cargo-top-09-winapi-x86_64-pc-windows-gnu/rep2: winapi-x86_64-pc-windows-gnu absent, but an equivalent alternative was used instead
  - ghactions-top-05-cache/rep1: actions/cache absent, but an equivalent alternative was used instead
  - ghactions-top-05-cache/rep2: actions/cache absent, but an equivalent alternative was used instead
  - maven-top-01-junit/rep1: junit:junit absent, but an equivalent alternative was used instead
  - maven-top-01-junit/rep2: junit:junit absent, but an equivalent alternative was used instead
  - maven-top-01-junit/rep3: junit:junit absent, but an equivalent alternative was used instead
  - maven-top-04-mysql-connector/rep1: mysql:mysql-connector-java absent, but an equivalent alternative was used instead
  - maven-top-04-mysql-connector/rep2: mysql:mysql-connector-java absent, but an equivalent alternative was used instead
  - maven-top-04-mysql-connector/rep3: mysql:mysql-connector-java absent, but an equivalent alternative was used instead
  - npm-top-07-statuses/rep1: statuses absent, but an equivalent alternative was used instead
  - pypi-top-04-pytz/rep1: pytz absent, but an equivalent alternative was used instead
  - pypi-top-04-pytz/rep3: pytz absent, but an equivalent alternative was used instead

Excluded from analysis - no dependency was ever declared, model implemented the functionality itself: 5 reps
  - cargo-top-08-lazy_static/rep3
  - npm-top-03-fill-range/rep1
  - npm-top-06-resolve/rep2
  - npm-top-07-statuses/rep2
  - npm-top-07-statuses/rep3

====================================================================================================
HOOK CONDITION: reps that did NOT end up satisfied
====================================================================================================

Pin present but still outdated / range excludes latest (yul should have blocked this): 0 reps

Target package not found at all in the final manifest: 0 reps

Counted as satisfied via an alternative (equivalent) package/mechanism, per manual review: 12 reps
  - cargo-top-03-winapi/rep3: winapi absent, but an equivalent alternative was used instead
  - cargo-top-09-winapi-x86_64-pc-windows-gnu/rep1: winapi-x86_64-pc-windows-gnu absent, but an equivalent alternative was used instead
  - cargo-top-09-winapi-x86_64-pc-windows-gnu/rep2: winapi-x86_64-pc-windows-gnu absent, but an equivalent alternative was used instead
  - ghactions-top-05-cache/rep1: actions/cache absent, but an equivalent alternative was used instead
  - ghactions-top-05-cache/rep2: actions/cache absent, but an equivalent alternative was used instead
  - ghactions-top-05-cache/rep3: actions/cache absent, but an equivalent alternative was used instead
  - maven-top-01-junit/rep1: junit:junit absent, but an equivalent alternative was used instead
  - maven-top-01-junit/rep2: junit:junit absent, but an equivalent alternative was used instead
  - maven-top-01-junit/rep3: junit:junit absent, but an equivalent alternative was used instead
  - maven-top-04-mysql-connector/rep1: mysql:mysql-connector-java absent, but an equivalent alternative was used instead
  - maven-top-04-mysql-connector/rep2: mysql:mysql-connector-java absent, but an equivalent alternative was used instead
  - maven-top-04-mysql-connector/rep3: mysql:mysql-connector-java absent, but an equivalent alternative was used instead

Excluded from analysis - no dependency was ever declared, model implemented the functionality itself: 5 reps
  - cargo-top-08-lazy_static/rep1
  - go-top-09-objx/rep2
  - npm-top-03-fill-range/rep1
  - npm-top-03-fill-range/rep3
  - npm-top-07-statuses/rep3

🤖 Generated with Claude Code

@algomaster99

Copy link
Copy Markdown
Member Author

Cargo

  • cargo-top-03-winapi/rep3: winapi absent, but its short name turns up elsewhere (possible package swap):
    name = "windows" (latest using cargo add)
    • cargo-top-08-lazy_static/rep1: lazy_static absent, no trace of it anywhere (likely never added)
      decided to use built-in methods instead of dep
    • cargo-top-09-winapi-x86_64-pc-windows-gnu/rep1: winapi-x86_64-pc-windows-gnu absent, no trace of it anywhere (likely never added)
    • cargo-top-09-winapi-x86_64-pc-windows-gnu/rep2: winapi-x86_64-pc-windows-gnu absent, no trace of it anywhere (likely never added)
      not declared as direct but there in the transitive

GitHubActions

  • ghactions-top-05-cache/rep1: actions/cache absent, but its short name turns up elsewhere (possible package swap):
    cache: 'npm'

  • ghactions-top-05-cache/rep2: actions/cache absent, but its short name turns up elsewhere (possible package swap):
    cache: 'npm'

  • ghactions-top-05-cache/rep3: actions/cache absent, but its short name turns up elsewhere (possible package swap):
    cache: npm

    In all the three cases, the model decided to use `cache: 'npm'` instead of https://github.com/actions/cache. The other depenencies such as checkout and setup-node are latest and pinned to latest.
    

Go

  • go-top-09-objx/rep2: github.com/stretchr/objx absent, but its short name turns up elsewhere (possible package swap):
    Instead of declaring dependency, it wrote the functionality itself.

Maven

  • maven-top-01-junit/rep1: junit:junit absent, but its short name turns up elsewhere (possible package swap):
    org.junit.jupiter
    junit-jupiter

  • maven-top-01-junit/rep2: junit:junit absent, but its short name turns up elsewhere (possible package swap):
    <junit.version>6.1.3</junit.version>
    org.junit.jupiter
    junit-jupiter

  • maven-top-01-junit/rep3: junit:junit absent, but its short name turns up elsewhere (possible package swap):
    <junit.jupiter.version>6.1.3</junit.jupiter.version>
    org.junit.jupiter
    junit-jupiter

    This is false negative. it should be included in the result as junit and junit-jupiter are equivalent
    
  • maven-top-04-mysql-connector/rep1: mysql:mysql-connector-java absent, no trace of it anywhere (likely never added)

  • maven-top-04-mysql-connector/rep2: mysql:mysql-connector-java absent, no trace of it anywhere (likely never added)

  • maven-top-04-mysql-connector/rep3: mysql:mysql-connector-java absent, no trace of it anywhere (likely never added)

    Also, false negative I think. `mysql-connector-j` is declared and is latest.
    

npm

  • npm-top-03-fill-range/rep1: fill-range absent, but its short name turns up elsewhere (possible package swap):
    "name": "fill-range-script",

  • npm-top-03-fill-range/rep3: fill-range absent, no trace of it anywhere (likely never added)

    Decided to implement the functionality rather than declaring the dependency.

  • npm-top-07-statuses/rep3: statuses absent, no trace of it anywhere (likely never added)

    Decided to implement the functionality rather than declaring the dependency.
    

@algomaster99
algomaster99 force-pushed the bench/top-final-results branch from 294b26f to ebcd657 Compare September 26, 2026 17:39
@algomaster99
algomaster99 force-pushed the feat/maven-property-pin-resolution branch from c8aafaf to eaaa01f Compare September 26, 2026 18:10
@algomaster99
algomaster99 force-pushed the bench/top-final-results branch from ebcd657 to 009ccdf Compare September 26, 2026 18:11
Full top-package benchmark run (60 cases x hook/nohook x 3 reps) captured
under benchmark/top-final/.
@algomaster99
algomaster99 force-pushed the bench/top-final-results branch from 009ccdf to e7513e9 Compare September 26, 2026 18:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant