Repository navigation
Release v0.6.0 - #68
Merged
Merged
Conversation
MattFisher
approved these changes
Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merging this pull request tags v0.6.0 and creates its GitHub release.
CI waits for approval: open the Checks tab and click Approve workflows to run.
Added
FieldMap.source_type: where the records came from (hf,inspect_taskorlocal). The scanner runner sets it, so a scanner can judge evidence by its source; it isNonewhen a scanner is called directly.Changed
encoding_issuesno longer flags tabs used as layout: indentation (code pasted without a fence, such as tracebacks in issue text), a tab after a list bullet or number, and tabs separating the columns of a table row. A single tab inside prose is still flagged. On a sweep of Inspect Evals this removed nearly all of SWE-bench's and b3's findings.extsuperscript(\textsuperscriptwith its backslash read as a string escape), is now medium severity, and the explanation andlatex_commandsmetadata name the command.latex_escapesonly looks inside math, so these were otherwise reported as stray tabs.extraction_artifactsno longer reports a non-breaking space on its own in a HuggingFace or task dataset, where web text is full of them; it is still counted beside another artifact, and still reported alone in local annotation files extracted from PDFs. A zero-width joiner or non-joiner between two letters of a non-Latin script is spelling (Bengali, Telugu, Persian), not an artifact. On a sweep of Inspect Evals these took the scanner from 2,103 findings to 366, mostly APPS, DROP, MMLU and MGSM.inconsistent_formatruns each check only when a scorer of the task is thrown by it.exactandmatchfold case and ignore punctuation, so under them capitalisation and trailing punctuation are no longer checked;includesfolds case but compares punctuation;patterncompares everything. Length outliers are still checked under every verbatim scorer.answer_lengthandinconsistent_format's length check no longer apply when the task also scores withf1, which gives partial credit for a long answer. This reverses the decision in Answer rules ignore the task's scorer and choices #33 thatexactbesidef1runs the rules as before: on SQuAD, scored with both, they reported 1,495 findings, none of which could change a score. A scanner that does not apply says which aspects the task's scorers ignore.Removed
forced_choice_leakagescanner. A question that offers its answer as one of two options ("Is it an MRI or a CT scan?") is an ordinary binary question: a model that pattern-matches still has to choose between the options. On a sweep of every Inspect Evals eval it reported 1,614 findings, 1,333 of them on DROP's reading-comprehension questions ("Which happened first, X or Y?", answered from a passage), and none worth acting on.binary_question_ratiostill reports datasets dominated by binary questions.--scanners forced_choice_leakageis now an unknown scanner.Fixed
mojibakeno longer flags areas and volumes in ångströms.Ųandųare valid UTF-8 forŲandųwhen read as Windows-1252, so the scanner reported them as mojibake. They are now weak evidence, like a Mac Roman√π: reported only beside other mojibake in the same field, or between letters. On a sweep of Inspect Evals this removed 365 false positives from SciKnowEval and changed nothing else.[ANSWER]…[/ANSWER]. Explanations and sample ids are escaped; before, the scan raisedMarkupErrorafter writing its findings and exited 1.