Skip to content

benchmark-pr: a --save run whose results push is refused (org fork) still reports success #1702

Open

Description

@ddxd

What happens. When the head of a PR is an organization fork, GitHub does not allow "Allow edits by maintainers", so the results commit that /benchmark --save makes cannot be pushed back. The "Commit saved results" step in .github/workflows/benchmark-pr.yml does:

git push origin HEAD:"${PR_BRANCH}" || echo "Warning: could not push to fork (permissions)"

The push fails with 403, the step prints the warning, the job ends green, and the PR comment looks the same as a saved run. Nothing on the PR says the results were not saved, so the PR can be merged without them.

Example. #1668 (mipyre, head leocaolab/HttpArena). Run 40687985062:

[detached HEAD a28d9e6a7] Benchmark results: mipyre  [skip ci]
remote: Permission to leocaolab/HttpArena.git denied to github-actions[bot].
fatal: unable to access 'https://github.com/leocaolab/HttpArena.git/': The requested URL returned error: 403
Warning: could not push to fork (permissions)

The run is marked success. #1668 was merged on 2026-10-31 with frameworks/mipyre/ but no site/data/results/mipyre.json.

Suggestion. When the push fails on a --save run, say so in the PR comment (for example "⚠️ results not saved: could not push to <fork> (403). Enable Allow edits by maintainers, or the maintainer can run the Benchmark workflow with save=true after merge") and mark the job failed, the same way the job is already flagged red when a framework's benchmark fails. Optionally, check maintainerCanModify before running a --save benchmark and refuse early, so the machine time is not spent on results that cannot be saved.

Activity

  1. added
    bugSomething isn't working
    platformEverything related to benchmark exection, including the PR workflow.
    on Nov 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingplatformEverything related to benchmark exection, including the PR workflow.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions