Skip to content

fix: reject NaN values in score validation - #1021

Closed
Gareth.Y (linjiaye929) wants to merge 1 commit into
braintrustdata:mainfrom
linjiaye929:codex/reject-nan-scores
Closed

Gareth.Y (linjiaye929) wants to merge 1 commit into
braintrustdata:mainfrom
linjiaye929:codex/reject-nan-scores

Conversation

@linjiaye929

Copy link
Copy Markdown
Contributor

Summary

Reject float("nan") at the existing score-validation boundaries. Scores are documented as numbers in [0, 1], with None representing a skipped score. The current score < 0 or score > 1 checks reject infinities but let NaN through because both comparisons are false.

Use the inclusive range predicate in Score.__post_init__ and the log-score validator. This also covers Score.from_dict / dict-scorer normalization. Valid scores, booleans, None, and the existing error messages are unchanged.

The generic JSON sanitizer intentionally converts nonfinite payload values into strings. This PR does not change that behavior: accepting NaN as a score previously allowed a scores mapping to reach that sanitizer and become {"scores": {"accuracy": "NaN"}}, instead of being rejected as an invalid numeric score. No backend behavior is asserted.

Tests

Python 3.14.7, using existing dependencies from the project's frozen test group:

  • Focused official tests before the fix: 2 failed, 23 passed, 2 subtests passed.
  • After the fix: 24 passed, 3 subtests passed. This includes the complete test_score.py and only the two new pure-validator test functions in test_logger.py; no logger or network request is constructed by those functions.
  • Existing generic JSON nonfinite-value control: 1 passed; NaN/Infinity remain stringified in arbitrary payloads.
  • Supplemental local reproduction covering the constructor, dict-scorer normalization and direct score validation: 4 tests passed after the fix.
  • Ruff check/format, changed-file Pylint --errors-only, git diff --check, and all seven full-repository pre-commit hooks pass. mise is unavailable in this environment, so the underlying pre-commit run --all-files command was run directly with the configured tool versions.

Runs used an external-socket guard and made no live Braintrust/provider calls. No dependencies or cassettes changed. The full core suite was not rerun: an earlier unrelated full-core invocation was blocked by the execution safety gate. Whole-SDK lint, static type checks, provider suites and other Python versions were not run.

AI assistance was used for analysis, implementation and local verification. No human approval or hosted CI success is claimed.

@linjiaye929
Gareth.Y (linjiaye929) marked this pull request as ready for review October 31, 2026 05:42
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-31T05:44:57.083587Z 22defbb Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants