[j-chim]

Several adapters leave temperature, max_tokens, sample counts and split empty although the source publishes them (harness dumps, eval logs, request logs, page data). This PR reads those values. It also removes two constants the sources do not state: HLE temperature 0.0 and MMLU-Pro "5-shot CoT".

Changes

hfopenllm_v2

• The Space API returns HTTP 500. The leaderboard table now comes from the open-llm-leaderboard/contents parquet at a pinned revision (--contents-parquet for offline runs; --source-api keeps the old path).
• Run settings come from the open-llm-leaderboard/results dumps at a pinned revision (--results-dir for offline runs), matched per (model, precision). A field is written only when every subtask and every dump for that pair report the same value; otherwise it is null and counted in the report.
• Written: output_type, num_fewshot (all six groups); temperature, max_tokens, do_sample (IFEval, MATH Lvl 5); samples_number, plus dataset_size when a run used a limit; hf_split; harness_dataset_path when the dump's dataset differs from hf_repo (IFEval, MATH); git_hash, transformers_version in eval_library.additional_details.
• evaluation_id now includes the precision. Before, 78 float16/bfloat16 pairs had the same id.

Inspect converter

• Generation args are read from model_generate_config.merge(plan.config), the config inspect_ai applies to each call (checked in 0.3.75, 0.3.90, 0.3.271). The converter read only model_generate_config, so settings defined on the task were null.
• If a plan step's params name temperature, max_tokens, top_p or top_k, or pass a model, that field is left null and listed in overridden_by_step. terminal_solver and generation_args_source are recorded.
• reasoning is set only when reasoning_effort is present (it was False when absent).
• samples_number is the number of sample ids; dataset_size is recorded when it differs. hf_split is read from task_args.split when it is train, val or test. eval_library.version is the inspect_ai version; other packages go to additional_details.

lm_eval converter: output_type is always written; generation_args is None for loglikelihood and multiple_choice tasks; samples_number is the effective n-samples count.

HELM converter: adapter_method written as given; hf_split set from the split already in details; duplicate sample_ids removed.

vals_ai

• If a model entry has a temperature or max_output_tokens value, it is used. If the entry value is null and the page's methodology section states one value for all models, that value is used (SAGE, MedCode, MedScribe: temperature 1, max_tokens 30000; MMMU: temperature 0). temperature_source and max_tokens_source say whether the value came from the row or the page. vals_methodology_url points at the section.
• The run fails if the quoted sentence is not found on the page.
• Also written from the entries: reasoning (boolean only), the harness name in agentic_eval_config, token totals and task results in score_details.details.
• Two routes that embed the same slug are converted once (reverse_eng now serves the srebench page).

arc_agi

• Clones arcprize/arc_agi_v1_public_eval and arc_agi_v2_public_eval at pinned commits (full no-checkout clone, sparse checkout of the leaderboard's model directories, history from the same clone). --arc-runs-dir and --arc-history-dir for offline runs.
• A score result is filled when the directory's task files and results.json reproduce the published score and cost at the row's precision, every attempt logs the same cap key and value, and no path was rewritten in the history. Written: max_tokens, cap_key, request_log, scored_tasks. Cost results and other datasets are not changed. A directory that fails a check is recorded as request_args_unavailable=<check>.

hle: temperature was 0.0 on every result. Scale's statement is "temperature 0.0 when configurable", and no row says whether its model was configurable. temperature is now null unless the row note matches "Sampled at temperature = , top_p = ." or "Sampled at temperature ". The statement is kept as source_generation_policy, the note as source_row_note. The judge prompt text no longer names a temperature.

mmlu_pro: prompt_style was "5-shot CoT" on every result. The leaderboard has no per-row shot count, and rows named "(0-shot)" contradict the label. prompt_style is now written only when the model name contains "(N-shot)".

Schema: the samples_number description is now "Number of items the reported score covers". eval_types.py is regenerated; only that description changes.

Rules applied

• A value is written only when the source's run record states it. No defaults, no inference.
• When records disagree, the field is left null and counted.
• Placeholders, the evaluation_id format and hf_repo values are unchanged.

Testing

• uv run pytest: 1581 passed, 44 skipped (core); 1642 passed, 2 skipped with the helm extra. ruff check clean.
• Offline: the hfopenllm dump index matches an independently built table (0 differences); the converted output matches the 4,574 existing datastore records except for the added fields. All 49 saved Vals pages convert and validate. The ARC join over local datastore rows fills the same 26 runs as the earlier analysis.
• Live runs into a scratch directory, all validator clean: hle 53 records; mmlu_pro 261; vals_ai 2,907 (the rsi_index page has no embedded data; the existing parser reports it); arc_agi 263 records in 24 s, 26 filled; hfopenllm_v2 4,574 from the pinned inputs.

Notes

• After these adapters run, the datastore holds both the old and the new records. Flat rebuild: mark superseded captures in the index #307 marks the old ones in the index.
• hfopenllm_v2 is not scheduled. It needs one manual full run, after which the 4,574 old records should be retired (the id prefix changed).
• CONTRIBUTING.md:126 says evaluation_id ends in retrieved_timestamp; fields.md:215 says never key it on now. This PR keeps the CONTRIBUTING format. Separate issue.

🤖 Generated with Claude Code