Bump the python group across 1 directory with 5 updates

[dependabot]

Bumps the python group with 5 updates in the / directory:

Package	From	To
openai	3.16.2	3.22.1
inspect-ai	0.3.265	0.3.273
inspect-evals	0.21.0	0.22.0
ruff	0.16.8	0.16.9
watchfiles	1.2.0	1.3.0

Updates openai from 3.16.2 to 3.22.1

Release notes

Sourced from openai's releases.

v3.22.1

3.22.1 (2026-09-30)

Bug Fixes

• api: correct the missing authentication error message (#3993) (51692ed), closes #3962
• transform NotRequired typed dictionary fields (#3995) (5544c18)

Chores

• tests: mark sample_file.txt as text with LF line endings (#3879) (fd78a48)

Documentation

• examples: fix two stale example comments (#3877) (1c4830b)
• examples: print event.delta in the streaming helpers doc (#3876) (063375b)

v3.22.0

3.22.0 (2026-09-29)

Features

• api: add computer use to beta agents (#3989) (f5dddb5)

v3.21.0

3.21.0 (2026-09-29)

Features

• api: add GPT-6.1 Sol model identifier (#3986) (160b2c0)

v3.20.0

3.20.0 (2026-09-28)

Features

• api: add Agents credential and session options (#3967) (bb68198)
• api: add Cyber access programs to Responses (#3956) (09c5b6f)
• responses: opt in to incremental WebSocket text and tool snapshots (#3973) (d0207b4)
• responses: preserve detailed WebSocket accumulator snapshots (#3981) (a380cf2)

Bug Fixes

... (truncated)

Changelog

Sourced from openai's changelog.

3.22.1 (2026-09-30)

Bug Fixes

• api: correct the missing authentication error message (#3993) (51692ed), closes #3962
• transform NotRequired typed dictionary fields (#3995) (5544c18)

Chores

• tests: mark sample_file.txt as text with LF line endings (#3879) (fd78a48)

Documentation

• examples: fix two stale example comments (#3877) (1c4830b)
• examples: print event.delta in the streaming helpers doc (#3876) (063375b)

3.22.0 (2026-09-29)

Features

• api: add computer use to beta agents (#3989) (f5dddb5)

3.21.0 (2026-09-29)

Features

• api: add GPT-6.1 Sol model identifier (#3986) (160b2c0)

3.20.0 (2026-09-28)

Features

• api: add Agents credential and session options (#3967) (bb68198)
• api: add Cyber access programs to Responses (#3956) (09c5b6f)
• responses: opt in to incremental WebSocket text and tool snapshots (#3973) (d0207b4)
• responses: preserve detailed WebSocket accumulator snapshots (#3981) (a380cf2)

Bug Fixes

• client: retry unmapped TLS transport errors (#3982) (0d35a26)
• live: avoid hangs at fractional transcript grouping deadlines (#3970) (4ef4129)
• live: keep query parameters out of WebSocket endpoint paths (#3972) (f9c458b)
• live: preserve caller queues and prevent uncertain WebSocket replay (#3980) (80e9686)

... (truncated)

Commits

• 841d929 release: 3.22.1 (#3994)
• 5544c18 fix: transform NotRequired typed dictionary fields (#3995)
• 1c4830b docs(examples): fix two stale example comments (#3877)
• fd78a48 chore(tests): mark sample_file.txt as text with LF line endings (#3879)
• 5559bdb test(api): cover generated WebSocket resources in redirect harness (#3992)
• 51692ed fix(api): correct the missing authentication error message (#3993)
• 063375b docs(examples): print event.delta in the streaming helpers doc (#3876)
• 5a4bb96 Fix false results for whitespace-padded true responses (#3854)
• 49e4366 release: 3.22.0 (#3990)
• f5dddb5 feat(api): add computer use to beta agents (#3989)
• Additional commits viewable in compare view

Updates inspect-ai from 0.3.265 to 0.3.273

Changelog

Sourced from inspect-ai's changelog.

0.3.273 (29 September 2026)

• Anthropic: Prompts can mark a stable prefix for caching while allowing the remaining text to change.
• OpenAI: GPT-5.6 and later support explicit prompt-cache boundaries; other models retain automatic caching.
• OpenAI: Support for GPT-6.1 Sol (gpt-6.1-sol), including its context window and output limits.
• Mistral: Support for mistralai 3.0, which is now the minimum required version.
• Logs: Reading samples with resolve_attachments is much faster for long conversations; in full mode, deeply nested model API call content may keep two more nesting levels.
• Bugfix: self_critique(), and model_graded_qa()/model_graded_fact() with model_role=None, now critique or grade with the correct model when one task is evaluated against several models.
• Reading a remote .eval log from a non-S3 filesystem (e.g. gs://, az://) no longer stalls other running work for the whole download.
• S3: Streaming uploads now read each multipart part from the source file in a single worker-thread call rather than 32 chunked calls, reducing event-loop wakeups during log flushes.
• Recording an eval or model error no longer spends seconds syntax highlighting its traceback: the stored ANSI traceback keeps its frames and source snippets, without syntax colouring.
• Bugfix: subprocess() and Docker sandbox exec() no longer intermittently fail with BrokenPipeError when the command exits before reading its input.

0.3.272 (28 September 2026)

• Anthropic: Support for Claude Sonnet 5.5 (claude-sonnet-5-5): reasoning_effort="none" turns off up-front thinking, forced tool choice degrades to auto, and computer use uses the computer toolset on the Claude API and Vertex.
• Tools: web_browser() is deprecated, logs a warning when called, and will be removed in a future release.
• Inspect View: Shard logs under <name>.shards/ are hidden once a successful merged log covers them (inspect view --show-shards to list them); bundles leave them out.
• Computer tool: click actions called without a coordinate now click at the current cursor position instead of failing, and the tool description states which actions require one.
• Computer tool: back_click and forward_click now work (with a rebuilt aisiuk/inspect-computer-tool image); previously they failed inside the container regardless of arguments.

0.3.271 (26 September 2026)

• CLI: New --extra-headers and --extra-body options for inspect eval and inspect eval-set, taking an inline JSON or YAML mapping.
• Google: Token counting falls back to a local estimate, with a warning, when the countTokens endpoint is unavailable or rejects the request.

0.3.270 (25 September 2026)

• eval_set: a task whose dataset GREW since its last run (a strict superset, with stable sample ids) is now topped up in place — the prior samples are reused and only the newly added samples run — instead of re-running the whole dataset. A shrunk dataset, or growth without stable ids, still triggers a full re-run.
• LiteLLM Proxy: GPT-5 and later, o-series and Codex models served by OpenAI use the Responses API by default, streamed, carrying reasoning between turns (-M responses_api=false to opt out).
• LiteLLM Proxy: Key and team aliases resolve to the model they route to, instead of failing for lack of model info.
• LiteLLM Proxy: Claude 4.7 and later models with a reasoning effort record summarized thinking in the log, including through older LiteLLM versions, which returned none (-M thinking_display=omitted to opt out).
• LiteLLM Proxy: Claude codename deployments that LiteLLM sends extended thinking fail with an error naming the proxy model_info flags that enable adaptive thinking.
• LiteLLM Proxy: web_search() without a search provider usable through the proxy fails when the model is first called, with an error naming the fix.
• Checkpointing: after a resume, restored events reach hooks, the live viewer, ACP clients and transcript readers with long text and images inline instead of as attachment:// references; raw model API calls stay condensed until the sample completes.
• Bugfix: f1() now counts repeated tokens (multiset overlap, matching SQuAD F1); scores can rise or fall for answers or targets containing repeated words.
• Bugfix: A sample retried or requeued in a shared log directory no longer shows the previous attempt's events alongside its own while it runs.

0.3.269 (25 September 2026)

• LiteLLM Proxy: New litellm-proxy provider for models served by a LiteLLM proxy, which reads each model's upstream model for context window, cost, reasoning and Claude prompt caching.
• CLI: Quoted YAML and JSON strings in --env arguments preserve commas as literal text (such as --env 'NO_PROXY="localhost,127.0.0.1"') instead of being coerced into lists. (#5368)
• DeepSeek: Support for DeepSeek-V4.1-Flash (deepseek-flash), including image input; model info notes that the retired deepseek-v4-flash and deepseek-v4-flash-vision-exp names are now served by V4.1 Flash.
• Sandbox agent bridge: host tools exposed with bridged_tools are again denied unless the model proposed the call in a bridged generation, once per proposal, with or without an approval policy (0.3.265 ran them regardless as a stopgap); BridgedToolsSpec(require_proposal=False) opts a server out.
• Bugfix: Docker sandboxes for samples a SampleSource adds (including an empty-seed task with sandbox="docker") no longer fail with a LookupError, and their containers and generated compose files are cleaned up at the end of the run.
• Bedrock: Models that return encrypted reasoning (OpenAI's GPT-5.6 family) no longer fail every request, and their reasoning is now carried across turns so multi-turn and tool-calling evals work, streamed or not.
• Scoring: match(), includes(), exact(), f1(), pattern() and answer() now record reason="no_response" when the raw model completion is empty or whitespace only, so a model that returned nothing is distinguishable from one that answered wrong. Score values are unchanged. (#5376)
• Grok: Non-streaming requests cut off by attempt_timeout are now retried, and ones cut off by a sample time_limit are recorded as that limit, instead of failing with a bare cancellation.
• Grok: Safety refusals that report "I can't help with that request" now produce a content-filter response instead of failing the sample.
• Bugfix: Samples whose solver is cut off by a cancellation that inspect did not issue are now recorded as sample errors instead of being scored as completed.

... (truncated)

Commits

• 9e44f1b update changelog for release
• b925c20 Scan shared messages through the declared ChatMessage schema (#5610)
• dd97900 Stop env tests leaking NO_PROXY into later tests (#5607)
• 8f93af4 Anthropic: cache a fixed prompt prefix without rewriting the varying tail on ...
• a5d76a0 Support for OpenAI GPT-6.1 Sol (#5609)
• 18acb82 Fix intermittent BrokenPipeError from subprocess() when the command exits bef...
• 0695989 ci-perf publisher: apply no auto label; a maintainer opts an issue in (#5511)
• 59b8b4d Recording an eval error no longer spends seconds syntax highlighting its trac...
• cafff0b Reduce event-loop wakeups during S3 log flushes: read each upload part in one...
• 15ec05c Fix reading a remote non-S3 .eval log (gs://, az://) stalling all other work ...
• Additional commits viewable in compare view

Updates inspect-evals from 0.21.0 to 0.22.0

Release notes

Sourced from inspect-evals's releases.

v0.22.0

Existing Evals

• PaperBench (v5-C): Criteria the judge could not grade are no longer scored as criteria the reproduction did not meet. They are left out of the weighted rubric average on both sides of the division, after bounded retries: judge replies are classified before use, so a refusal, an empty reply, or a file ranking that names nothing readable is retried rather than parsed as an essay or graded against no files, and a grading prompt that overflows the context window is retried once with half the file budget before erroring the sample. A rubric with no gradeable criterion returns Score.unscored(), and failures of the machinery around the judge error the sample. Score.value is now a dict with keys score and graded_weight_fraction, the second reporting how much of the rubric was graded. The score mean can move in either direction: per-sample scores rise when a judge reply was ungradeable, and samples whose grader call raised now leave the denominator.

• AgentBench (v4-B): Add sandbox_type/sandbox_config provider configuration following the cross-eval sandbox-config pattern (#1115).

• PaperBench (v3-C): Add a task-level sandbox_config: SandboxEnvironmentSpec override for non-default providers (#1115).

• Mind2Web-SC (v3-B): Load the full 200-sample dataset instead of truncating to the first 10 samples via hardcoded slice, and disambiguate sample IDs across ALLOW/DENY variants sharing the same task annotation.

• HLE: original_accuracy now follows CAIS's rounded percentage calculation using the selected dataset size after dataset filters and before run limits. The dataset size is recorded in sample metadata, so omitted questions and missing judgments remain in the denominator for static and rolling datasets. This metric requires single-attempt results and dataset-size metadata; other metric calculations are unchanged.

• HLE: the default run config now evaluates the HLE-Verified gold subset (only_hle_verified_gold: true); original.yaml keeps the full CAIS dataset. Default-run scores are not comparable with earlier versions.

• HLE: select metrics and their arguments through the scorer spec's metrics field in run configs. Metric factories resolve through Inspect registration. Score calculations are unchanged, but each shipped config now selects its own metrics: the default run reports hle/accuracy, hle/stderr, hle/cerr and hle/unscored and no longer emits hle/original_accuracy; original.yaml reports hle/original_accuracy, hle/stderr, hle/cerr and hle/unscored. Direct use of the HLE judge factories now requires metric selection on the containing task. Scorer and metric names in the spec are Inspect registry names, and everything HLE defines is registered under an eval-qualified name (inspect_evals/hle/regex_judge, inspect_evals/hle/accuracy, ...), so the score columns are now hle/regex_judge and hle/regex_judge1 (hle/json_judge under original.yaml) and every metric row carries the hle/ prefix.

• PaperBench (v4-B): Update blacklist URL monitor to identify scp-like SSH clones git clone git@github.com:owner/repo.git. Deviates from original paper implementation, however, as the paper implementation outlines that the blacklist monitor is used in addition to human checking, this change is helps with the human check and aligns with the intent of the paper.

• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (v4-A), PAWS: Paraphrase Adversaries from Word Scrambling (v4-A), MaCBench: Probing the limitations of multimodal language models for chemistry and materials research (v2-A): A completion that does not match the answer pattern is now scored INCORRECT with reason="invalid_response_format", rather than NOANSWER. This follows pattern()/answer() upstream in inspect_ai#4629, which attributes an unmatched completion to the model under test. Headline metrics are unaffected: value_to_float maps both NOANSWER and INCORRECT to 0.0, so accuracy and stderr are unchanged on every run. What moves is where the non-answer is recorded, from the score value to Score.reason; log analysis that counts NOANSWER values should read reason instead.

• BBH: Challenging BIG-Bench Tasks (v3-A), ChemBench: Are large language models superhuman chemists? (v3-B), ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (v3-A) and WorldSense: Grounded Reasoning Benchmark (v3-A): The same pattern()/answer() change as above. It reaches BBH's binary-choice subsets, ChemBench's multiple-choice questions, infinite_bench_code_run and all of WorldSense; the other scorers in those evals are unchanged. Headline metrics are unaffected for the same reason.

• KernelBench: Use a standalone uv project to pin sandbox dependencies independently of the host lockfile, so host dependency updates do not require rebuilding the published image.

• KernelBench: Remove 34 host-only packages, including inspect-ai, from the sandbox image while preserving all 148 retained package versions. The pinned default image is now the reduced 2026-09-17 build, validated on a Tesla T4 through the unchanged runner.

• KernelBench: Add the kernelbench_sandbox_check maintenance task, which runs the eval's own scorer over fixture kernels with known verdicts to verify a sandbox image on GPU hardware, plus an opt-in pytest wrapper and a Hawk example.

• DS-1000: Add the ds1000_sandbox_check maintenance task, which runs the eval's scorer over fixture solutions with known verdicts inside the pinned sandbox image and checks the pinned packages are installed; accuracy 1.0 certifies the image.

• WorldSense (v4-A): Derive the sample id from the dataset's own unique Key. The previous id, (tuple_ID, problemname), is shared by every trial of a tuple, so filter_duplicate_ids kept one trial per tuple and dropped 46,872 of the 87,048 trials. Because the kept trial is the first in file order, the evaluated set held no sample whose gold answer was FALSE, IMPOSSIBLE or 2, while POSSIBLE rose from 23.1% to 50.0% of the set and TRUE from 15.4% to 33.3%.

• StereoSet: Measuring stereotypical bias in pretrained language models (v4-A): Use the dataset's own per-record id as the sample id. The previous id hashed (context, target, bias_type), which several records share while presenting different candidate sentences and gold labels, so filter_duplicate_ids silently dropped 8 of 2,123 intersentence records and 40 of 2,106 intrasentence records.

• SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models (v4-B): Build the sample id from every field that distinguishes a record (question, domain, task, subtask, choices, answer key, answer) hashed to 16 hex digits. The previous id hashed only (question, domain, task), which 3,394 records share with another record while differing in choices, answer or subtask, so filter_duplicate_ids silently dropped them and the full task evaluated 66,386 of 70,196 rows. It now evaluates 69,736; the 460 rows still removed are identical to another row on every field.

• MMLU: Measuring Massive Multitask Language Understanding (v4-A): Include the answer key in MMMLU sample ids. Four pairs of translated rows (FR_FR, IT_IT and two in AR_XY) have identical question text and options but different answers because the translation collapsed two distinct English questions, and the previous id dropped the second row of each pair as a duplicate. Both rows are now evaluated, and the id is hashed to 16 hex digits because the all-languages default config has 196,588 rows. The 5-shot ids also include the subject, because the few-shot examples depend on it, so the 78 questions listed under two subjects become two prompts in mmlu_5_shot (14,015 samples) while mmlu_0_shot keeps one copy (13,937).

• BBH: Challenging BIG-Bench Tasks: Exclude the two unscorable ruin_names rows by id through drop_known_broken, with the upstream report (suzgunmirac/BIG-Bench-Hard#19) recorded beside each id, instead of matching the question text inside the prompt. The same two rows are excluded as before, so results and the task version are unchanged. The README gains a "Known dataset issues" section and dataset_samples is corrected from 250 to the 6,509 samples the default task loads across all subsets.

Other

• Fix docker_handling="force_build" for digest-pinned sandbox images. Local builds use the image tag; default and forced-pull runs retain the configured digest.

• Evaluation checks now use inspect-evals-lint 0.2.1, installed through the dev dependency group. Run uv run inspect-evals-lint <eval_name>. The shared utils package is linted as a helper alongside the evaluations. Existing unresolved model roles remain warnings through an allowlist.

• Add scorers_from_spec, metrics_from_spec and with_metrics to inspect_evals.utils, so a task can take its scorer and metric selection as run-config data (a scorer spec dict with args, graders and metrics) and resolve it through Inspect's own loaders. Extracted from the HLE metric configuration work as a first step towards upstreaming to Inspect.

... (truncated)

Changelog

Sourced from inspect-evals's changelog.

[0.22.0] — 2026-09-25

Existing Evals

• PaperBench (v5-C): Criteria the judge could not grade are no longer scored as criteria the reproduction did not meet. They are left out of the weighted rubric average on both sides of the division, after bounded retries: judge replies are classified before use, so a refusal, an empty reply, or a file ranking that names nothing readable is retried rather than parsed as an essay or graded against no files, and a grading prompt that overflows the context window is retried once with half the file budget before erroring the sample. A rubric with no gradeable criterion returns Score.unscored(), and failures of the machinery around the judge error the sample. Score.value is now a dict with keys score and graded_weight_fraction, the second reporting how much of the rubric was graded. The score mean can move in either direction: per-sample scores rise when a judge reply was ungradeable, and samples whose grader call raised now leave the denominator.

• AgentBench (v4-B): Add sandbox_type/sandbox_config provider configuration following the cross-eval sandbox-config pattern (#1115).

• PaperBench (v3-C): Add a task-level sandbox_config: SandboxEnvironmentSpec override for non-default providers (#1115).

• Mind2Web-SC (v3-B): Load the full 200-sample dataset instead of truncating to the first 10 samples via hardcoded slice, and disambiguate sample IDs across ALLOW/DENY variants sharing the same task annotation.

• HLE: original_accuracy now follows CAIS's rounded percentage calculation using the selected dataset size after dataset filters and before run limits. The dataset size is recorded in sample metadata, so omitted questions and missing judgments remain in the denominator for static and rolling datasets. This metric requires single-attempt results and dataset-size metadata; other metric calculations are unchanged.

• HLE: the default run config now evaluates the HLE-Verified gold subset (only_hle_verified_gold: true); original.yaml keeps the full CAIS dataset. Default-run scores are not comparable with earlier versions.

• HLE: select metrics and their arguments through the scorer spec's metrics field in run configs. Metric factories resolve through Inspect registration. Score calculations are unchanged, but each shipped config now selects its own metrics: the default run reports hle/accuracy, hle/stderr, hle/cerr and hle/unscored and no longer emits hle/original_accuracy; original.yaml reports hle/original_accuracy, hle/stderr, hle/cerr and hle/unscored. Direct use of the HLE judge factories now requires metric selection on the containing task. Scorer and metric names in the spec are Inspect registry names, and everything HLE defines is registered under an eval-qualified name (inspect_evals/hle/regex_judge, inspect_evals/hle/accuracy, ...), so the score columns are now hle/regex_judge and hle/regex_judge1 (hle/json_judge under original.yaml) and every metric row carries the hle/ prefix.

• PaperBench (v4-B): Update blacklist URL monitor to identify scp-like SSH clones git clone git@github.com:owner/repo.git. Deviates from original paper implementation, however, as the paper implementation outlines that the blacklist monitor is used in addition to human checking, this change is helps with the human check and aligns with the intent of the paper.

• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (v4-A), PAWS: Paraphrase Adversaries from Word Scrambling (v4-A), MaCBench: Probing the limitations of multimodal language models for chemistry and materials research (v2-A): A completion that does not match the answer pattern is now scored INCORRECT with reason="invalid_response_format", rather than NOANSWER. This follows pattern()/answer() upstream in inspect_ai#4629, which attributes an unmatched completion to the model under test. Headline metrics are unaffected: value_to_float maps both NOANSWER and INCORRECT to 0.0, so accuracy and stderr are unchanged on every run. What moves is where the non-answer is recorded, from the score value to Score.reason; log analysis that counts NOANSWER values should read reason instead.

• BBH: Challenging BIG-Bench Tasks (v3-A), ChemBench: Are large language models superhuman chemists? (v3-B), ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (v3-A) and WorldSense: Grounded Reasoning Benchmark (v3-A): The same pattern()/answer() change as above. It reaches BBH's binary-choice subsets, ChemBench's multiple-choice questions, infinite_bench_code_run and all of WorldSense; the other scorers in those evals are unchanged. Headline metrics are unaffected for the same reason.

• KernelBench: Use a standalone uv project to pin sandbox dependencies independently of the host lockfile, so host dependency updates do not require rebuilding the published image.

• KernelBench: Remove 34 host-only packages, including inspect-ai, from the sandbox image while preserving all 148 retained package versions. The pinned default image is now the reduced 2026-09-17 build, validated on a Tesla T4 through the unchanged runner.

• KernelBench: Add the kernelbench_sandbox_check maintenance task, which runs the eval's own scorer over fixture kernels with known verdicts to verify a sandbox image on GPU hardware, plus an opt-in pytest wrapper and a Hawk example.

• DS-1000: Add the ds1000_sandbox_check maintenance task, which runs the eval's scorer over fixture solutions with known verdicts inside the pinned sandbox image and checks the pinned packages are installed; accuracy 1.0 certifies the image.

• WorldSense (v4-A): Derive the sample id from the dataset's own unique Key. The previous id, (tuple_ID, problemname), is shared by every trial of a tuple, so filter_duplicate_ids kept one trial per tuple and dropped 46,872 of the 87,048 trials. Because the kept trial is the first in file order, the evaluated set held no sample whose gold answer was FALSE, IMPOSSIBLE or 2, while POSSIBLE rose from 23.1% to 50.0% of the set and TRUE from 15.4% to 33.3%.

• StereoSet: Measuring stereotypical bias in pretrained language models (v4-A): Use the dataset's own per-record id as the sample id. The previous id hashed (context, target, bias_type), which several records share while presenting different candidate sentences and gold labels, so filter_duplicate_ids silently dropped 8 of 2,123 intersentence records and 40 of 2,106 intrasentence records.

• SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models (v4-B): Build the sample id from every field that distinguishes a record (question, domain, task, subtask, choices, answer key, answer) hashed to 16 hex digits. The previous id hashed only (question, domain, task), which 3,394 records share with another record while differing in choices, answer or subtask, so filter_duplicate_ids silently dropped them and the full task evaluated 66,386 of 70,196 rows. It now evaluates 69,736; the 460 rows still removed are identical to another row on every field.

• MMLU: Measuring Massive Multitask Language Understanding (v4-A): Include the answer key in MMMLU sample ids. Four pairs of translated rows (FR_FR, IT_IT and two in AR_XY) have identical question text and options but different answers because the translation collapsed two distinct English questions, and the previous id dropped the second row of each pair as a duplicate. Both rows are now evaluated, and the id is hashed to 16 hex digits because the all-languages default config has 196,588 rows. The 5-shot ids also include the subject, because the few-shot examples depend on it, so the 78 questions listed under two subjects become two prompts in mmlu_5_shot (14,015 samples) while mmlu_0_shot keeps one copy (13,937).

• BBH: Challenging BIG-Bench Tasks: Exclude the two unscorable ruin_names rows by id through drop_known_broken, with the upstream report (suzgunmirac/BIG-Bench-Hard#19) recorded beside each id, instead of matching the question text inside the prompt. The same two rows are excluded as before, so results and the task version are unchanged. The README gains a "Known dataset issues" section and dataset_samples is corrected from 250 to the 6,509 samples the default task loads across all subsets.

Other

• Fix docker_handling="force_build" for digest-pinned sandbox images. Local builds use the image tag; default and forced-pull runs retain the configured digest.

• Evaluation checks now use inspect-evals-lint 0.2.1, installed through the dev dependency group. Run uv run inspect-evals-lint <eval_name>. The shared utils package is linted as a helper alongside the evaluations. Existing unresolved model roles remain warnings through an allowlist.

• Add scorers_from_spec, metrics_from_spec and with_metrics to inspect_evals.utils, so a task can take its scorer and metric selection as run-config data (a scorer spec dict with args, graders and metrics) and resolve it through Inspect's own loaders. Extracted from the HLE metric configuration work as a first step towards upstreaming to Inspect.

... (truncated)

Commits

• 41c72ea Release v0.22.0 (#2543)
• 054cad9 feat(utils): make filter_duplicate_ids acknowledgement optional with a deprec...
• b3ed58b refactor(bbh): exclude the two broken ruin_names rows by id, with the report ...
• 265f31c feat(utils): make filter_duplicate_ids refuse to drop non-identical samples (...
• 61fdf46 HLE: configurable metrics, CAIS original_accuracy, gold subset default (7-E) ...
• 2329ee2 docs: guidance on reporting dataset problems to dataset maintainers (#2527)
• 154f59a fix(sciknoweval): hash every distinguishing field into the sample id (#2525)
• 5f2013d fix(stereoset): use the dataset's own record id as the sample id (#2524)
• 2cb54ae fix(worldsense): identify a trial by its unique Key, not by its tuple (#2507)
• 9f77276 fix(paperbench): pass reason= to Score.unscored instead of the metadata key (...
• Additional commits viewable in compare view

Updates ruff from 0.16.8 to 0.16.9

Release notes

Sourced from ruff's releases.

0.16.9

Release Notes

Released on 2026-09-24.

Preview features

• [ruff] Avoid false positives for overloaded division (RUF069) (#28309)

Bug fixes

• [flake8-bugbear] Avoid false positives for calls with keyword arguments (B009, B010, B043) (#28776)
• [flake8-tidy-imports] Allow lazy imports to be used in deferred annotations (TID255) (#28767)

Rule changes

• Update LibCST-based fixes for Python 3.15 (#28616)
• [flake8-pyi] Mention stubs in the diagnostic message (PYI002) (#28542)

Documentation

• Fix horizontal overflow on the rules documentation page (#28699)
• Update rules table with category information (#28651)
• [flake8-annotations] Clarify that ANN401 checks return types in addition to arguments (#28334)
• [flake8-bugbear] Document type-checker interaction (B010) (#28509)
• [flake8-comprehensions] Document map/generator exception behavior (C417) (#27794)
• [ruff] Mention related isort settings (RUF022) (#28719)

Contributors

• @​qinpei-dev
• @​sanjayrohith
• @​ntBre
• @​webdevsamran
• @​zaniebot
• @​ewdurbin
• @​MichaReiser
• @​spaceone
• @​IbrahimKhan12
• @​devtechedge
• @​GruffElixir

Install ruff 0.16.9

Install prebuilt binaries via shell script

curl --proto '=https' --tlsv1.2 -LsSf https://releases.astral.sh/github/ruff/releases/download/0.16.9/ruff-installer.sh | sh

... (truncated)

Changelog

Sourced from ruff's changelog.

0.16.9

Released on 2026-09-24.

Preview features

• [ruff] Avoid false positives for overloaded division (RUF069) (#28309)

Bug fixes

• [flake8-bugbear] Avoid false positives for calls with keyword arguments (B009, B010, B043) (#28776)
• [flake8-tidy-imports] Allow lazy imports to be used in deferred annotations (TID255) (#28767)

Rule changes

• Update LibCST-based fixes for Python 3.15 (#28616)
• [flake8-pyi] Mention stubs in the diagnostic message (PYI002) (#28542)

Documentation

• Fix horizontal overflow on the rules documentation page (#28699)
• Update rules table with category information (#28651)
• [flake8-annotations] Clarify that ANN401 checks return types in addition to arguments (#28334)
• [flake8-bugbear] Document type-checker interaction (B010) (#28509)
• [flake8-comprehensions] Document map/generator exception behavior (C417) (#27794)
• [ruff] Mention related isort settings (RUF022) (#28719)

Contributors

• @​qinpei-dev
• @​sanjayrohith
• @​ntBre
• @​webdevsamran
• @​zaniebot
• @​ewdurbin
• @​MichaReiser
• @​spaceone
• @​IbrahimKhan12
• @​devtechedge
• @​GruffElixir

Commits

• 0be08a2 Bump version to 0.16.9 (#28882)
• b4920b7 Rename ruff_cli to ruff_command_line (#28881)
• 47c751b Update dependency astral-sh/uv to v0.12.18 (#28880)
• 8c244e5 [flake8-comprehensions] Document map/generator exception behavior (C417...
• 5edf5a1 Use target form in rooster.version_files (#28876)
• 915bb2b [ty] Prefer existing @ paths over response files in Ruff and ty (#28877)
• 4710e1a ci(github): update version number in placeholder of issue template (#28871)
• eedfc62 [ty] Propagate outer type context through cast calls (#28855)
• ceaa6a0 [ty] Contain rendered code within Markdown fences (#28869)
• dba0f30 authorize ruff-pre-commit dispatch via OIDC (#28867)
• Additional commits viewable in compare view

Updates watchfiles from 1.2.0 to 1.3.0

Release notes

Sourced from watchfiles's releases.

v1.3.0 2026-09-21

What's Changed

• Bump pyo3 from 0.28.3 to 0.29.0 by @​dependabot[bot] in samuelcolvin/watchfiles#382
• Add Python 3.14t to Windows wheel builds by @​1998LJ in samuelcolvin/watchfiles#393
• Build abi3 wheels for CPython by @​samuelcolvin in samuelcolvin/watchfiles#396
• prep v1.3.0 release by @​samuelcolvin in samuelcolvin/watchfiles#397

New Contributors

• @​1998LJ made their first contribution in samuelcolvin/watchfiles#393

Full Changelog: samuelcolvin/watchfiles@v1.2.0...v1.3.0

Commits

• e74b8fc prep v1.3.0 release (#397)
• 026c971 Build abi3 wheels for CPython (#396)
• 59099d2 Add Python 3.14t to Windows wheel builds (#393)
• 42257e8 Bump pyo3 from 0.28.3 to 0.29.0 (#382)
• See full diff in compare view

[dependabot]

Looks like these dependencies are updatable in another way, so this is no longer needed.