Repository navigation
fix(evaluations): warn once when a scorer returns an invalid score - #199
Merged
Merged
Conversation
The first invalid score from each scorer in a run now logs one console warning. The warning names the scorer and the value. The criterion result is still an invalid_score ERROR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
jeffdupont
approved these changes
Oct 30, 2026
Merged
jeffdupont
pushed a commit
that referenced
this pull request
Oct 30, 2026
🤖 I have created a release *beep* *boop* --- <details><summary>@launchdarkly/ai-claude-agents: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-claude-agents-0.5.0...@launchdarkly/ai-claude-agents-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **client:** add result variant and value to the feature_flag span event ([#186](#186)) * **SEC-9507:** enable Claude native tools only when the active variation attaches them * **claude-agents:** honour the config's max_tokens and disable thinking unless the config asks ([#183](#183)) * **skills:** rename FDv2SkillStore to LaunchDarklySkillStore ([#172](#172)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ### Features * **client:** add result variant and value to the feature_flag span event ([#186](#186)) ([8dbe2ec](8dbe2ec)) * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) * **skills:** give Claude Agents runs the skills attached to their variation ([#136](#136)) ([df16ba0](df16ba0)) * **skills:** rename FDv2SkillStore to LaunchDarklySkillStore ([#172](#172)) ([d3e7c5f](d3e7c5f)) ### Bug Fixes * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ([c6c00fc](c6c00fc)) * **claude-agents:** honour the config's max_tokens and disable thinking unless the config asks ([#183](#183)) ([f58ce21](f58ce21)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) * **SEC-9507:** dispatch litellm-messages tool calls only to tools offered for the request ([6926e8a](6926e8a)) * **SEC-9507:** enable Claude native tools only when the active variation attaches them ([6926e8a](6926e8a)) * **SEC-9507:** pass handlers only the tools attached to the active variation ([6926e8a](6926e8a)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-claude-messages: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-claude-messages-0.5.0...@launchdarkly/ai-claude-messages-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ### Features * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ([c6c00fc](c6c00fc)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-langchain-agents: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-langchain-agents-0.5.0...@launchdarkly/ai-langchain-agents-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **client:** add result variant and value to the feature_flag span event ([#186](#186)) * **AIC-3439:** let wildcard handlers accept only named providers ([#111](#111)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ### Features * **AIC-3439:** let wildcard handlers accept only named providers ([#111](#111)) ([9371975](9371975)) * **client:** add result variant and value to the feature_flag span event ([#186](#186)) ([8dbe2ec](8dbe2ec)) * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3592:** make native LangGraph hand-offs complete ([#192](#192)) ([025f2ce](025f2ce)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ([c6c00fc](c6c00fc)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-langchain-messages: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-langchain-messages-0.5.0...@launchdarkly/ai-langchain-messages-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **client:** add result variant and value to the feature_flag span event ([#186](#186)) * **AIC-3439:** let wildcard handlers accept only named providers ([#111](#111)) * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) ### Features * **AIC-3439:** let wildcard handlers accept only named providers ([#111](#111)) ([9371975](9371975)) * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) ([15a0d37](15a0d37)) * **client:** add result variant and value to the feature_flag span event ([#186](#186)) ([8dbe2ec](8dbe2ec)) * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3726:** realign three usage IDs with the 1.0 factory names ([#197](#197)) ([fc83c7a](fc83c7a)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-litellm-agents: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-litellm-agents-0.2.0...@launchdarkly/ai-litellm-agents-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. ### Features * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-litellm-messages: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-litellm-messages-0.2.0...@launchdarkly/ai-litellm-messages-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **SEC-9507:** enable Claude native tools only when the active variation attaches them * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ### Features * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ([c6c00fc](c6c00fc)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) * **SEC-9507:** dispatch litellm-messages tool calls only to tools offered for the request ([6926e8a](6926e8a)) * **SEC-9507:** enable Claude native tools only when the active variation attaches them ([6926e8a](6926e8a)) * **SEC-9507:** pass handlers only the tools attached to the active variation ([6926e8a](6926e8a)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-node: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-node-0.4.0...@launchdarkly/ai-node-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **evaluations:** promote Evaluations from Code to core ([#181](#181)) * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) * **skills:** rename FDv2SkillStore to LaunchDarklySkillStore ([#172](#172)) ### Features * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) ([15a0d37](15a0d37)) * **evaluations:** promote Evaluations from Code to core ([#181](#181)) ([676a2ed](676a2ed)) * **skills:** rename FDv2SkillStore to LaunchDarklySkillStore ([#172](#172)) ([d3e7c5f](d3e7c5f)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-openai-agents: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-openai-agents-0.5.0...@launchdarkly/ai-openai-agents-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **client:** add result variant and value to the feature_flag span event ([#186](#186)) * **AIC-3687:** rename createOpenAIAgentHandler to createOpenAIAgentsHandler ([#174](#174)) ### Features * **AIC-3687:** rename createOpenAIAgentHandler to createOpenAIAgentsHandler ([#174](#174)) ([cfba4d2](cfba4d2)) * **client:** add result variant and value to the feature_flag span event ([#186](#186)) ([8dbe2ec](8dbe2ec)) * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3726:** realign three usage IDs with the 1.0 factory names ([#197](#197)) ([fc83c7a](fc83c7a)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-openai-messages: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-openai-messages-0.5.0...@launchdarkly/ai-openai-messages-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ### Features * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) ([15a0d37](15a0d37)) * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ([c6c00fc](c6c00fc)) * **AIC-3726:** realign three usage IDs with the 1.0 factory names ([#197](#197)) ([fc83c7a](fc83c7a)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-otel: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-otel-0.3.0...@launchdarkly/ai-otel-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** spans carry launchdarkly.project_id instead of highlight.project_id ### Bug Fixes * **client:** stamp launchdarkly.project_id on the telemetry resource ([#176](#176)) ([93fb412](93fb412)) </details> <details><summary>@launchdarkly/ai-server: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-server-0.5.0...@launchdarkly/ai-server-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. * **client:** add result variant and value to the feature_flag span event ([#186](#186)) * **AIC-3439:** let wildcard handlers accept only named providers ([#111](#111)) * **SEC-9507:** enable Claude native tools only when the active variation attaches them * **evaluations:** export each offline row as one trace ([#175](#175)) * **evaluations:** promote Evaluations from Code to core ([#181](#181)) * **evaluations:** remove default pass rate threshold ([#184](#184)) * **client:** spans carry launchdarkly.project_id instead of highlight.project_id * **AIC-3687:** rename createOpenAIAgentHandler to createOpenAIAgentsHandler ([#174](#174)) * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) * **skills:** rename FDv2SkillStore to LaunchDarklySkillStore ([#172](#172)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ### Features * **AIC-3439:** let wildcard handlers accept only named providers ([#111](#111)) ([9371975](9371975)) * **AIC-3686:** rename the LangChain and OpenAI messages factories to *MessagesHandler ([#173](#173)) ([15a0d37](15a0d37)) * **AIC-3687:** rename createOpenAIAgentHandler to createOpenAIAgentsHandler ([#174](#174)) ([cfba4d2](cfba4d2)) * **client:** add result variant and value to the feature_flag span event ([#186](#186)) ([8dbe2ec](8dbe2ec)) * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) * **client:** point judge spans at the run they graded ([#200](#200)) ([ba9c936](ba9c936)) * **evaluations:** export each offline row as one trace ([#175](#175)) ([05a7eb1](05a7eb1)) * **evaluations:** promote Evaluations from Code to core ([#181](#181)) ([676a2ed](676a2ed)) * **evaluations:** remove default pass rate threshold ([#184](#184)) ([f3a3e37](f3a3e37)) * **skills:** give Claude Agents runs the skills attached to their variation ([#136](#136)) ([df16ba0](df16ba0)) * **skills:** rename FDv2SkillStore to LaunchDarklySkillStore ([#172](#172)) ([d3e7c5f](d3e7c5f)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **AIC-3598:** type toolHandlers on the handler contract; declare @langchain/anthropic ([#170](#170)) ([c6c00fc](c6c00fc)) * **AIC-3601:** stop multi-edge routing text from asking a router for a final answer ([#177](#177)) ([10cce54](10cce54)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) * **client:** stamp launchdarkly.project_id on the telemetry resource ([#176](#176)) ([93fb412](93fb412)) * **evaluations:** update stale test import ([#189](#189)) ([f4d8083](f4d8083)) * **evaluations:** warn once when a scorer returns an invalid score ([#199](#199)) ([f43c888](f43c888)) * **graph:** report a node's own judge scores on its node_done stream event ([#182](#182)) ([1230f02](1230f02)) * **SEC-9507:** dispatch litellm-messages tool calls only to tools offered for the request ([6926e8a](6926e8a)) * **SEC-9507:** enable Claude native tools only when the active variation attaches them ([6926e8a](6926e8a)) * **SEC-9507:** pass handlers only the tools attached to the active variation ([6926e8a](6926e8a)) </details> <details><summary>@launchdarkly/ai-typesafe: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-typesafe-0.2.0...@launchdarkly/ai-typesafe-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. ### Features * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-vercel-agents: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-vercel-agents-0.3.0...@launchdarkly/ai-vercel-agents-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. ### Features * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> <details><summary>@launchdarkly/ai-vercel-messages: 1.0.0</summary> ## [1.0.0](https://github.com/launchdarkly/js-ai-sdk/compare/@launchdarkly/ai-vercel-messages-0.3.0...@launchdarkly/ai-vercel-messages-1.0.0) (2026-10-30) ### ⚠ BREAKING CHANGES * **client:** conversation content is now captured on spans by default. To keep it off, call configure({ options: { captureContent: false } }) once at startup. * **client:** createHandler returns a new wrapper function rather than adding providesFor to the function you pass and returning that same function. Use the handler createHandler returns; the function you passed in no longer carries providesFor. ### Features * **client:** capture conversation content on spans by default and add configure() ([d9c7fc3](d9c7fc3)) ### Bug Fixes * **AIC-3573:** never send a NativeTool to the model as a function tool ([#194](#194)) ([ec43b85](ec43b85)) * **client:** return a new handler from createHandler instead of changing the function passed in ([d9c7fc3](d9c7fc3)) ### Dependencies * The following workspace dependencies were updated * dependencies * @launchdarkly/ai-server bumped from ^0.5.0 to ^1.0.0 </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > This **release-please** PR cuts **1.0.0** for every workspace package (`@launchdarkly/ai-server`, `@launchdarkly/ai-node`, provider adapters, `@launchdarkly/ai-otel`, `@launchdarkly/ai-typesafe`) by updating `.release-please-manifest.json`, each `package.json` / `src/version.ts`, and adding **1.0.0** changelog sections. Provider packages bump their `@launchdarkly/ai-server` dependency to **^1.0.0**. > > The release notes bundled here flag **breaking** upgrades integrators must handle: **`configure()`** with **conversation content on spans by default** (opt out via `captureContent: false`); **`createHandler`** now returns a **new wrapper** instead of mutating the passed function; renamed factories (**`*MessagesHandler`**, **`createOpenAIAgentsHandler`**); **`LaunchDarklySkillStore`** (was `FDv2SkillStore`); **Evaluations promoted to core** with offline export/trace and threshold behavior changes; telemetry **`launchdarkly.project_id`** (was `highlight.project_id`); richer **`feature_flag`** span events; and **SEC-9507** tool scoping (native tools and handlers only see tools on the active variation). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit f87c9a6. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This change matches launchdarkly/python-ai-sdk#92.
The first time a scorer returns an invalid score in a run, the SDK logs one
console.warn. The warning names the scorer and the value:An invalid score is a bool, a non-finite number, a number out of range, or a value that is not a number.
The criterion result does not change. It is still an
invalid_scoreERROR event, and the run continues. The SDK already rejects a boolean score, so this PR changes only the warning.Why
An invalid score is recorded as a per-criterion ERROR, so the problem is easy to miss. A caller who upgrades with scorers that still return booleans sees only ERROR results in LaunchDarkly. The warning shows the cause in the caller's own logs. The SDK logs once for each scorer in each run, so a large dataset does not flood the log.
Tests
vitest runinpackages/client: 2082 passed, 11 skipped.yarn typecheckandbiome check .pass.🤖 Generated with Claude Code
Note
Overview
When a custom scorer returns a non-numeric, out-of-range, or otherwise invalid value, the evaluation phase now emits a
console.warnthe first time each scorer does so within a run, naming the scorer and the validation message (aligned with the Python SDK).Criterion behavior is unchanged: each row still gets an
invalid_scoreERROR event and the run continues. A per-runSetdedupes warnings across rows so large datasets do not spam logs; a new run can warn again.Tests cover one warning per bad scorer across multiple rows, repeat warnings on subsequent runs, and silence for valid scores.
Reviewed by Cursor Bugbot for commit b716232. Bugbot is set up for automated code reviews on this repo. Configure here.