Skip to content

Add ExtractBench Pro leaderboard and result submissions - #83

Draft
xujustinj wants to merge 1 commit into
mainfrom
codex/pro-leaderboard
Draft

xujustinj wants to merge 1 commit into
mainfrom
codex/pro-leaderboard

Conversation

@xujustinj

Copy link
Copy Markdown

ExtractBench Pro has published scores but no leaderboard CSV or documented way to submit an update in this repository. This adds a Pro table directly alongside the original ExtractBench tables and gives contributors a result-submission workflow.

  • Add leaderboard-pro.csv with the twelve published launch systems: overall results and all six challenge slices, using the 30-minute limit. Include perfect-document percentage, Value F1, box/page grounding F1, scoring denominators, and cost per page.
  • Document the CSV format, metric definitions, launch snapshot, and evidence needed for new or updated results. Add a benchmark-result PR template.
  • Generate both benchmarks' README tables with the existing script and check synchronization in CI. Original ExtractBench scores and generated tables are unchanged.

Validation

  • Python 3.12: uv run pytest tests/ -q — 902 passed, 29 skipped (optional runner dependencies).
  • uv run ruff format --check and uv run ruff check.
  • uv run python scripts/update_readme.py --check and git diff --check.
  • Compared every Pro CSV score and cost against the published launch results; verified twelve complete systems, seven slices per system, and the 100-document / 81-grounding-document overall denominators.
  • Verified the original CSV and both original generated README blocks are unchanged.

Three new tests catch challenge rows entering the overall ranking or incorrect tie-breaking, unreported grounding/cost being displayed as zero, and stale generated tables passing the check or being overwritten in check mode. No model inference was run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant