Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
b3d668c
hfopenllm_v2: read the leaderboard table and the harness dumps at pin…
j-chim Sep 28, 2026
da950d8
converters: read generation args from the merged config; record items…
j-chim Sep 28, 2026
a773b65
vals_ai: write null generation settings from the page's methodology s…
j-chim Sep 28, 2026
b590176
arc_agi: write max_tokens and task counts from the published public-e…
j-chim Sep 28, 2026
5f72006
hle: remove the fixed temperature 0.0; read temperature from the row …
j-chim Sep 28, 2026
2091386
mmlu_pro: write prompt_style only from the model name
j-chim Sep 28, 2026
231e8d2
flat rebuild: mark superseded captures in the index
j-chim Sep 28, 2026
f7aa9c3
catalog: hfopenllm_v2 inputs and arc_agi note
j-chim Sep 28, 2026
a1eef58
Merge remote-tracking branch 'origin/main' into pr307-refresh
j-chim Oct 21, 2026
a9f4e73
Merge remote-tracking branch 'origin/main' into pr306-refresh
j-chim Oct 21, 2026
18f66b2
Require result identity coverage for automatic supersession
j-chim Oct 21, 2026
f6a91d3
Read Vals page defaults and complete table payloads
j-chim Oct 21, 2026
202e25e
Keep Inspect sample counts per scorer and bound plan fills
j-chim Oct 21, 2026
1571a19
Expose published HF and ARC sample counts in uncertainty
j-chim Oct 21, 2026
822b3a0
Merge refreshed adapter fixes into supersession branch
j-chim Oct 21, 2026
6fc6a39
Reject Vals loader payloads without table fields
j-chim Oct 21, 2026
3be95e9
Merge Vals payload validation fix
j-chim Oct 21, 2026
3c570b1
Fetch pinned HF results in one Git pack
j-chim Oct 21, 2026
7efdba5
Merge HF results bulk fetch
j-chim Oct 21, 2026
0c90645
Read published flat aggregates from a pinned Git pack
j-chim Oct 21, 2026
1a13369
Backfill all inherited identities from bulk snapshots
j-chim Oct 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
300 changes: 274 additions & 26 deletions every_eval_ever/adapters/arc_agi/adapter.py

Large diffs are not rendered by default.

635 changes: 635 additions & 0 deletions every_eval_ever/adapters/arc_agi/run_logs.py

Large diffs are not rendered by default.

13 changes: 11 additions & 2 deletions every_eval_ever/adapters/catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,11 @@ def _helm(key: str, leaderboard: str, weekday: int) -> AdapterSpec:
notes=(
'Repointed 2026-08-12 to the JSON files behind '
'arcprize.org/leaderboard; the old '
'/media/data/leaderboard/evaluations.json endpoint is gone.'
'/media/data/leaderboard/evaluations.json endpoint is gone. '
'Also clones the two arcprize public-eval log datasets (full '
'no-checkout clones, about 130 MB of packs, then a sparse '
'checkout at the pinned commits); a live run measured 24 s, so '
'the default timeout stands.'
),
),
AdapterSpec(
Expand Down Expand Up @@ -315,14 +319,19 @@ def _helm(key: str, leaderboard: str, weekday: int) -> AdapterSpec:
weekday=4,
timeout_minutes=45,
runnable=False,
with_packages=('pyarrow',),
unrunnable_reason=(
'the Open LLM Leaderboard is no longer maintained upstream '
'(space discussion 1135); the archive is frozen, so a scheduled '
'refresh has nothing new to fetch'
),
notes=(
'The adapter still works for a one-off manual conversion of the '
'frozen archive (around 4,576 models).'
'frozen archive (4,576 model and precision rows): it reads the '
'leaderboard table from the open-llm-leaderboard/contents '
'parquet and the harness dumps from open-llm-leaderboard/results, '
'both at pinned revisions (pyarrow, from with_packages, or `uv run --with pyarrow`). The Space API is '
'no longer served and is only read with --source-api.'
),
),
AdapterSpec(
Expand Down
Loading
Loading