Repository navigation
Conversation
j-chim
changed the title
…ned revisions The Space API returns HTTP 500. The leaderboard table now comes from the open-llm-leaderboard/contents parquet at a pinned revision (--contents-parquet for offline runs; --source-api keeps the old path). Run settings come from the open-llm-leaderboard/results dumps at a pinned revision (--results-dir for offline runs), matched per (model, precision). A field is written only when every subtask and every dump for that pair report the same value; otherwise it is left null and counted in the report. Fields written: output_type and num_fewshot on all six groups; temperature, max_tokens and do_sample on IFEval and MATH Lvl 5; samples_number, plus dataset_size when a run used a limit; hf_split; harness_dataset_path when the dump's dataset differs from hf_repo (IFEval, MATH); git_hash and transformers_version in eval_library.additional_details. evaluation_id now includes the precision. Before, 78 float16/bfloat16 pairs had the same id. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… scored Inspect: generation args are read from model_generate_config.merge(plan.config), which is the config inspect_ai applies to each call. The converter read only model_generate_config, so settings defined on the task were null. If a plan step's params name temperature, max_tokens, top_p or top_k, or pass a model, that field is left null and listed in overridden_by_step. terminal_solver and generation_args_source are recorded. reasoning is set only when reasoning_effort is present (it was False when absent). samples_number is the number of sample ids, with dataset_size recorded when it differs. hf_split is read from task_args.split when it is train, val or test. eval_library.version is the inspect_ai version; other packages go to additional_details. lm_eval: output_type is always written. generation_args is None for loglikelihood and multiple_choice tasks. samples_number is the effective n-samples count, with dataset_size when the original count differs. HELM: adapter_method is written as given. hf_split is set from the split already in details. Duplicate sample_ids are removed. Schema: the samples_number description is now "Number of items the reported score covers". eval_types.py is regenerated; only that description changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tatement If a model entry has a temperature or max_output_tokens value, it is used. If the entry value is null and the page's methodology section states one value for all models, that value is used: SAGE, MedCode and MedScribe temperature 1 and max_tokens 30000; MMMU temperature 0. temperature_source and max_tokens_source record whether the value came from the row or from the page. vals_methodology_url points at the section. The run fails if the quoted sentence is not found in the page's methodology section. Also written from the entries: reasoning (only when it is a boolean), the harness name in agentic_eval_config.additional_details, and token totals and task results in score_details.details. Two routes that embed the same slug are converted once (reverse_eng now serves the srebench page). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…val logs The adapter clones arcprize/arc_agi_v1_public_eval and arc_agi_v2_public_eval at pinned commits (git clone --no-checkout, sparse checkout of the leaderboard's model directories, git log from the same clone). --arc-runs-dir and --arc-history-dir replace the clone in offline runs. A score result is filled when the directory's task files and results.json reproduce the published score and cost at the row's precision, every attempt logs the same cap key and value, and no path in the directory was rewritten in the history. Written: generation_args.max_tokens, cap_key, request_log and scored_tasks. Cost results and other datasets are not changed. A directory that fails a check is recorded as request_args_unavailable=<check> on the record and in the report. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…notes temperature was 0.0 on every result. Scale's statement is "temperature 0.0 when configurable", and no row says whether its model was configurable. temperature is now null unless the row note matches one of two sentence forms: "Sampled at temperature = <n>, top_p = <n>." or "Sampled at temperature <n>". The statement is written as source_generation_policy and the note as source_row_note. The judge prompt text no longer names a temperature. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
prompt_style was "5-shot CoT" on every result. The leaderboard has no per-row shot count, and rows named "(0-shot)" contradict the label. prompt_style is now written only when the model name contains "(N-shot)". Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A corrected or re-scraped record is published as a new file and the old
file stays, so consumers read both. Rows in entries.jsonl now have
evaluation_id, retrieved_timestamp, cron_adapter, cron_run_date,
result_count, superseded_by and superseded_reason.
flat/retired.jsonl ({object_uuid, reason, replaced_by?}) is applied first;
retired rows are excluded from the rule below. For rows with
type_of_addition=cron, rows with the same (collection, cron_adapter,
evaluation_id without its last segment) form a series when that segment
equals retrieved_timestamp. Every row except the newest gets superseded_by
= the newest row's uuid and superseded_reason = newer_capture.
A series is reported and not marked when two rows share a cron_run_date, or
when the newest row has fewer results than an older row. A collection that
still has rows without evaluation_id is skipped. --bootstrap-ids downloads
the aggregate objects once with snapshot_download to fill the new fields;
later runs fill up to 500 rows each. data/ and flat/objects/ are not
modified.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
hfopenllm_v2 lists pyarrow in with_packages (needed to read the contents parquet) and its note names the pinned inputs. The arc_agi note describes the clones; a live run took 24 s, so the default timeout is kept. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
j-chim
force-pushed
the
feat/flat-index-supersession
branch
from
September 28, 2026 19:40
84ba333 to
231e8d2
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #306. This branch includes its 7 commits plus one new commit,
flat rebuild: mark superseded captures in the index. Review that commit.A corrected or re-scraped record is published as a new file and the old file stays, so consumers read both. This PR marks the old file in the flat index.
data/andflat/objects/are not modified.Changes
entries.jsonlgainevaluation_id,retrieved_timestamp,cron_adapter,cron_run_date,result_count,superseded_by,superseded_reason.flat/retired.jsonl({object_uuid, reason, replaced_by?}) is applied first. Retired rows are excluded from the rule below. A uuid not in the index is reported, not fatal.type_of_addition: cron: rows with the same (collection,cron_adapter,evaluation_idwithout its last segment) form a series when that segment equalsretrieved_timestamp. Every row except the newest getssuperseded_by= the newest row's uuid andsuperseded_reason=newer_capture.cron_run_date(ambiguous_series) or when the newest row has fewer results than an older row (regressed_series). A collection that still has rows withoutevaluation_idis skipped.--bootstrap-idsdownloads the aggregate objects once withsnapshot_downloadto fill the new fields; later runs fill up to 500 rows each.entries_sha256. A rebuild with no changes commits nothing.superseded_by is null.Why the rule applies only to cron rows
Measured on the current datastore: HELM writes one record per scenario subject under one id prefix; the converters end the id with the eval's own timestamp; terminal-bench-2.0 publishes two records with one prefix in one cron run. A rule based on the id prefix alone would mark unrelated records as superseded.
Needs agreement
This is a structural change, opened as a draft per CONTRIBUTING.
tools/build_flat_datastore.pyin the datastore repo must write the same fields and apply the rules in the same order, or the two builders produce different entries hashes.Testing
tests/test_flat_rebuild.py: 124 tests (series and ties, cron scoping, tail check, ambiguous and regressed series, pending backfill, retire list including stale and dangling entries, single-download bootstrap, idempotent rebuild, no-op rebuild).🤖 Generated with Claude Code