Findings: scan the dataset through the eval's task

[MattFisher]

Dataset scans now read the eval's samples through its task by default, instead of scanning eval.yaml's HuggingFace asset with auto-detected arguments.

Why. A survey of 32 inspect_evals tasks ran inspect-dataset scan <hf asset> exactly as the adapter does. Only 3 ran, and all three scanned train, which none of those evals uses. The other 29 failed on:

Cause	Evals
Config name missing	12
Column names not auto-detected	7
No train split	4
google/ or openai/ owner mistaken for a Python package (inspect_dataset#31)	4
Script-based datasets	2

inspect-dataset scan inspect_evals/<task> worked for all 32.

Task scans. The adapter runs inspect-dataset scan inspect_evals/<task> for the first task in eval.yaml, or for a task declared in the pilot config. inspect-dataset then loads the samples through the eval's own loader, with its split, config, revision, field mapping and sample ids. Because that imports the eval, the command runs in the checkout's own environment: uv run --project <root> --frozen --with <inspect-dataset> inspect-dataset. --frozen means the checkout's uv.lock is never rewritten. INSPECT_AUDIT_DATASET_TASK_CMD overrides the command, with {ie_root} standing for the checkout path.

Declarations. A declaration with HuggingFace settings (path, config, split, revision, fields) still scans the HuggingFace dataset directly, as before. task cannot be combined with those settings; validation rejects it.

Recording. The run's subject dataset is eval.yaml's HuggingFace asset, or the task spec when there is none. The adapter does not use the summary's dataset fields in task mode, because a task scan's summary names the spec and a train split whatever the task loads (inspect_dataset#40). inputs.dataset gains mode and task. The Inputs section says "through the task's own loader".

StereoSet. Its HuggingFace declaration is removed from pilot.yaml in its own commit, so it scans through its task. The result is 18 findings (the real duplicate contexts) instead of 2,169. The sample ids are the same 2,123 in both modes, so those 18 findings keep their fingerprints. The config comment now shows how to declare a task.

Trial on real inputs (live inspect_evals checkout, pinned inspect-dataset afbc94c0b509):

Eval	Before	Now (task scan)
ARC	skip	12 findings
GSM8K	skip	4 findings
MMLU-Pro	skip	850 findings
HumanEval	skip	156 findings
StereoSet	2,169 findings	18 findings

The checkout had no tracked changes afterwards.

Limits.

• One task per eval: ARC, WMDP and LAB-Bench cover their first task unless a task is declared.
• A task scan imports the eval's code on the host, as running the eval would. That is fine for inspect_evals, but it needs a rethink before the adapter points at third-party repos.
• Many task-scan findings are still scanner misfires (letter targets, choices ignored, whole prompts split on "or"). Those are tracked as inspect_dataset issues #30 and #32–#39.
• The adapter does not yet read the scanner_status added by inspect_dataset#29. That can follow once #29 merges and the pin moves.

Gate: pre-commit clean, basedpyright 0, 501 passed.

🤖 Generated with Claude Code