[Raphael Fakhri (RaphaelFakhri)]

Summary

Fixes several problems with how error_score_handler is called when a row's task or scorers fail.

• Unhandled scores leaked across rows. _run_evaluator_internal_impl kept unhandled_scores in the enclosing function and updated it with nonlocal from every row. A row whose task raised read whatever the last finished row stored there. If an earlier row finished all its scorers, the value was None, so the handler never ran. If an earlier row had a failing scorer, the failed row only got an error score for that one scorer. With default_error_score_handler, failed rows dropped out of the score averages instead of counting as 0. unhandled_scores is now local to each row, which matches the JavaScript SDK.
• A raising handler aborted the whole eval. The handler was called while building the EvalResult, outside the row's try/except, so an exception from it cancelled every remaining row. It is now caught and reported with eprint, and the row keeps its other results without error scores.
• A mutating handler could corrupt later rows. The handler received the evaluator-wide scorer_names list itself. A handler that changed its argument changed which scorers ran on every later row. The handler now gets a copy.
• Error scores were logged after the root span ended. The handler ran after with root_span: exited, so the scores default_error_score_handler logs to the root span were written after metrics.end. It now runs inside the span block.

Tests

Three tests in test_framework.py, sharing an _error_score_evaluator builder:

• test_error_score_handler_gets_all_scorers_for_each_failed_row runs a scorer-error row, a task-error row, a clean row and another task-error row. It uses a handler that empties its list, and checks that each row's handler gets exactly the scorers that row is missing and that every row's scores are correct.
• test_error_score_handler_exception_does_not_abort_eval checks that a raising handler leaves the other rows scored.
• test_error_score_handler_logs_before_root_span_ends reads the raw memory-logger writes and checks that the error scores come before the root span's metrics.end.

Each test fails against framework.py from before its fix and passes after it.

cd py
python -m pytest src/braintrust/test_framework.py -k error_score_handler
# 3 passed
python -m pytest src/braintrust/test_framework.py src/braintrust/test_framework2.py src/braintrust/test_workflow_eval.py
# 187 passed, 4 skipped
nox -s test_core
# 985 passed, 69 skipped, 12 xfailed

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-14T20:14:12.916754Z	4b5e887	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "Codex (@codex) review" or "Codex (@codex) security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Gareth.Y (linjiaye929)]

Raphael Fakhri (@RaphaelFakhri), I found an adjacent entry-point issue while checking the error-score handler: EvalAsync accepts error_score_handler but omits it from its _EvalCommon(...) call. This reproduces with a single row on main (457aaa8):

import asyncio
from braintrust import Eval, EvalAsync
from braintrust.framework import default_error_score_handler


def task(input):
    raise ValueError("task failed")


def quality(output):
    return 1.0


kwargs = dict(
    name="handler-repro",
    data=[{"input": 1}],
    task=task,
    scores=[quality],
    error_score_handler=default_error_score_handler,
    no_send_logs=True,
    enable_cache=False,
)
print(Eval(**kwargs).results[0].scores)                    # {'quality': 0}
print(asyncio.run(EvalAsync(**kwargs)).results[0].scores)  # {}

A possible follow-up is forwarding error_score_handler=error_score_handler in that call. This single-row omission is independent of the per-row state issue and remains outside the current diff.

AI-assisted investigation; reproduced locally without backend calls.

[Raphael Fakhri (RaphaelFakhri)]

Good catch. That's a separate omission in a different entry point, so I'd keep this PR scoped to the per-row state. A one-line follow-up that forwards error_score_handler from EvalAsync makes sense on its own.

[Abhijeet Prasad (AbhiPrasad)]

made some changes, but thanks Raphael Fakhri (@RaphaelFakhri)!

[Abhijeet Prasad (AbhiPrasad)]

released with https://github.com/braintrustdata/braintrust-sdk-python/releases/tag/py-sdk-v0.46.1 - thanks again Raphael Fakhri (@RaphaelFakhri)