[Bin (xlb3055)]

Summary

Eval and EvalAsync accept zero-argument callable data factories through EvalData, but the runner only invokes objects recognized by inspect.isfunction or inspect.isroutine. A factory instance implementing __call__ is instead iterated and raises TypeError, aborting the evaluation before any case runs. functools.partial factories also fail on Python 3.12.

Use callable to recognize data factories, preserving Dataset iteration as the existing workflow evaluator does. Public signatures are unchanged.

Regression Coverage

Ten parameterized cases exercise both public evaluation APIs with a callable instance, functools.partial, a function, an iterator, and a list. They check single factory invocation, case order, task outputs, and scores.

• Python 3.12.10 baseline: 4 failed, 6 passed. After the fix: 10 passed.
• Python 3.14.6 baseline: 2 failed, 8 passed. After the fix: 10 passed. Python 3.14 already recognizes partial objects through inspect.isroutine.
• Python 3.14.6 nox -s test_core: 1131 passed, 106 skipped, 12 xfailed.
• Configured pre-commit checks pass for both changed files and across the full repository.
• Focused errors-only pylint 4.1.2 passes for both changed modules with their imports installed.

The new tests use local data/tasks/scorers, no_send_logs=True, and disabled caching. No model downloads or live provider requests are needed. The full provider/Python CI matrix and full nox pylint session have not been run locally.

Related to #917 only in the data-resolution area: that PR resolves awaitable data; this change recognizes synchronous callable factories and does not add awaitable handling.

AI assistance was used for investigation, implementation, tests, and this description.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-28T13:36:49.017063Z We couldn’t complete this review. Comment @codex review to try again.	07affa5	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "Codex (@codex) review" or "Codex (@codex) security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Abhijeet Prasad (AbhiPrasad)]

Hey Bin (@xlb3055) thanks for the PR! What was the motivation behind this change?

[Bin (xlb3055)]

Thanks for asking. The concrete failure is with a zero-argument callable data factory object, which EvalData allows: Eval(..., data=Factory()) currently tries to iterate the Factory instance and raises TypeError before running any case, even though Factory.__call__() returns the data. A functools.partial factory has the same failure on Python 3.12. This matters when an evaluation constructs its cases lazily or keeps configuration in a callable object.

The change recognizes those callable factories while leaving Dataset objects on the existing iteration path, matching the handling in workflow_eval.py. The regression tests exercise both Eval and EvalAsync with callable objects, partials, functions, iterators, and lists. I can narrow the scope if the intended EvalData contract excludes callable objects.

[Abhijeet Prasad (AbhiPrasad)]

Bin (@xlb3055) I’m more interested how you ran into this. How are you using braintrust evals in your projects?

[Bin (xlb3055)]

Thanks for clarifying, Abhijeet Prasad (@AbhiPrasad). This issue came from an AI-assisted review of the SDK's evaluation data-loading paths while exploring open-source evaluation tooling, followed by a local reproducer. It was not reported from a production Braintrust integration.

The starting point was the mismatch between the callable factory accepted by EvalData and the narrower inspect.isfunction / inspect.isroutine check in the runner. A small callable object returning local test cases reproduced the error through both Eval and EvalAsync. The tests use local tasks and scorers with logging disabled; they do not depend on a hosted evaluation project or model calls.

My earlier reply explained the technical failure and potential uses of callable factories, but did not answer your question about how this was encountered. The example here is a local regression case, not a claim of an existing deployed workflow. I hope that makes the contribution's context clearer.