Skip to content

feat(evaluations)!: remove default pass rate threshold - #238

Merged
aknight-ld merged 1 commit into
mainfrom
AIC-3697-remove-default-pass-rate-threshold
Oct 28, 2026
Merged

feat(evaluations)!: remove default pass rate threshold#238
aknight-ld merged 1 commit into
mainfrom
AIC-3697-remove-default-pass-rate-threshold

Conversation

@aknight-ld

@aknight-ld aknight-ld commented Oct 28, 2026 •
edited by cursor Bot
Loading

Copy link
Copy Markdown
Contributor

Directly undoes the work of #226 prior to GA. Set threshold default to none until server is available to handle criterion thresholds.

Note

Overview
Reverts the client-side 0.95 default for pass_rate_threshold on Judge and Scorer (undoing #226 ahead of GA). Both types now default pass_rate_threshold to None, and passRateThreshold is omitted from evaluation criteria wire payloads unless the caller sets it explicitly.

Per-row threshold behavior is unchanged (judges still default to 0.5 on the wire). Tests and the experimental evaluations API report were updated to match the slimmer default options objects.

Reviewed by Cursor Bugbot for commit 1734294. Bugbot is set up for automated code reviews on this repo. Configure here.

@aknight-ld
aknight-ld requested a review from a team as a code owner October 28, 2026 17:49
@aknight-ld
aknight-ld merged commit 882d099 into main Oct 28, 2026
11 checks passed
@aknight-ld
aknight-ld deleted the AIC-3697-remove-default-pass-rate-threshold branch October 28, 2026 20:47
@github-actions github-actions Bot mentioned this pull request Oct 28, 2026
Merged
jeffdupont pushed a commit that referenced this pull request Oct 30, 2026
@github-actions
chore: release main (#231)
7aacbba 
🤖 I have created a release *beep* *boop*
---


<details><summary>launchdarkly-ai-server: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-server-0.5.0...launchdarkly-ai-server-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **client:** init_client raises when the LaunchDarkly client did not
initialize
([#254](#254))
* **evaluations:** route generation and judges to a scoped wildcard
handler only for its named providers, and accept two scoped wildcards of
one mode when their lists differ
* **AIC-3439:** let wildcard handlers accept only named providers
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
* **evaluations:** promote Evaluations from Code to core
([#235](#235))
* **client:** capture conversation content by default
([#213](#213))
* **evaluations:** remove default pass rate threshold
([#238](#238))
* **evaluations:** set default pass rate threshold
([#226](#226))
* **client:** spans carry launchdarkly.project_id instead of
highlight.project_id
* **AIC-3687:** rename create_openai_agent_handler to
create_openai_agents_handler
([#225](#225))
* **skills:** rename FDv2SkillStore to LaunchDarklySkillStore
([#223](#223))

### Features

* **AIC-3439:** let wildcard handlers accept only named providers
([454a4de](454a4de))
* **AIC-3687:** rename create_openai_agent_handler to
create_openai_agents_handler
([#225](#225))
([a7cdbc0](a7cdbc0))
* **client:** add result variant and value to the feature_flag span
event ([#240](#240))
([040d487](040d487))
* **client:** capture conversation content by default
([#213](#213))
([77a7cf5](77a7cf5))
* **client:** point judge spans at the run they graded
([#253](#253))
([863d423](863d423))
* **evaluations:** promote Evaluations from Code to core
([#235](#235))
([0bc43a5](0bc43a5))
* **evaluations:** remove default pass rate threshold
([#238](#238))
([882d099](882d099))
* **evaluations:** route generation and judges to a scoped wildcard
handler only for its named providers, and accept two scoped wildcards of
one mode when their lists differ
([454a4de](454a4de))
* **evaluations:** set default pass rate threshold
([#226](#226))
([7ee368c](7ee368c))
* **skills:** rename FDv2SkillStore to LaunchDarklySkillStore
([#223](#223))
([11f2c0b](11f2c0b))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3573:** never send a NativeTool to the model as a function tool
([#248](#248))
([73caeae](73caeae))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))
* **AIC-3601:** stop graph routing text inviting a router to answer for
its target
([#230](#230))
([27b8d09](27b8d09))
* **client:** init_client raises when the LaunchDarkly client did not
initialize
([#254](#254))
([6e66c00](6e66c00))
* **client:** stamp launchdarkly.project_id on the telemetry resource
([#229](#229))
([5347558](5347558))
* **evaluations:** copy inline tool schemas and name both sources on a
key collision
([#227](#227))
([bfbdd4c](bfbdd4c))
* **evaluations:** send criterion errors as errorMessage and skip
criteria for rows that did not generate
([#228](#228))
([8b05b56](8b05b56))
* **graph:** report a node's own judge scores on its node_done stream
event ([#236](#236))
([2f3f9ee](2f3f9ee))


### Documentation

* **AIC-3692:** say which project id attribute the telemetry resource
carries
([#239](#239))
([64eff43](64eff43))
* state that Agent Skills and Evaluations from Code need 1.0.0
([#191](#191))
([fe3ef08](fe3ef08))
</details>

<details><summary>launchdarkly-ai-python: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-python-0.4.0...launchdarkly-ai-python-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
* **evaluations:** promote Evaluations from Code to core
([#235](#235))

### Features

* **evaluations:** promote Evaluations from Code to core
([#235](#235))
([0bc43a5](0bc43a5))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))


### Documentation

* state that Agent Skills and Evaluations from Code need 1.0.0
([#191](#191))
([fe3ef08](fe3ef08))
</details>

<details><summary>launchdarkly-ai-claude-agents: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-claude-agents-0.3.0...launchdarkly-ai-claude-agents-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
* **claude-agents:** honour the config's max_tokens and disable thinking
unless the config asks
([#237](#237))
* **client:** capture conversation content by default
([#213](#213))
* **AIC-3685:** make the internal provider helpers private
([#224](#224))
* **skills:** rename FDv2SkillStore to LaunchDarklySkillStore
([#223](#223))

### Features

* **AIC-3685:** make the internal provider helpers private
([#224](#224))
([9afdb94](9afdb94))
* **client:** add result variant and value to the feature_flag span
event ([#240](#240))
([040d487](040d487))
* **client:** capture conversation content by default
([#213](#213))
([77a7cf5](77a7cf5))
* **skills:** rename FDv2SkillStore to LaunchDarklySkillStore
([#223](#223))
([11f2c0b](11f2c0b))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))
* **claude-agents:** honour the config's max_tokens and disable thinking
unless the config asks
([#237](#237))
([a7ded1b](a7ded1b))
</details>

<details><summary>launchdarkly-ai-claude-messages: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-claude-messages-0.3.0...launchdarkly-ai-claude-messages-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
* **client:** capture conversation content by default
([#213](#213))

### Features

* **client:** add result variant and value to the feature_flag span
event ([#240](#240))
([040d487](040d487))
* **client:** capture conversation content by default
([#213](#213))
([77a7cf5](77a7cf5))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3573:** never send a NativeTool to the model as a function tool
([#248](#248))
([73caeae](73caeae))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))
* **AIC-3735:** send Claude Messages sampling parameters through
extra_body
([#256](#256))
([c6d0376](c6d0376))
</details>

<details><summary>launchdarkly-ai-openai-agents: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-openai-agents-0.3.0...launchdarkly-ai-openai-agents-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
* **client:** capture conversation content by default
([#213](#213))
* **AIC-3685:** make the internal provider helpers private
([#224](#224))
* **AIC-3687:** rename create_openai_agent_handler to
create_openai_agents_handler
([#225](#225))

### Features

* **AIC-3685:** make the internal provider helpers private
([#224](#224))
([9afdb94](9afdb94))
* **AIC-3687:** rename create_openai_agent_handler to
create_openai_agents_handler
([#225](#225))
([a7cdbc0](a7cdbc0))
* **client:** add result variant and value to the feature_flag span
event ([#240](#240))
([040d487](040d487))
* **client:** capture conversation content by default
([#213](#213))
([77a7cf5](77a7cf5))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3573:** never send a NativeTool to the model as a function tool
([#248](#248))
([73caeae](73caeae))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))
* **AIC-3716:** build OpenAI native graph node tools with FunctionTool
([#251](#251))
([b859b67](b859b67))
* **AIC-3726:** realign three usage IDs with the 1.0 factory names
([#252](#252))
([1c4d904](1c4d904))
</details>

<details><summary>launchdarkly-ai-vercel-agents: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-vercel-agents-0.2.0...launchdarkly-ai-vercel-agents-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))

### Features

* **client:** add result variant and value to the feature_flag span
event ([#240](#240))
([040d487](040d487))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3573:** never send a NativeTool to the model as a function tool
([#248](#248))
([73caeae](73caeae))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))
</details>

<details><summary>launchdarkly-ai-litellm-agents: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-litellm-agents-0.2.0...launchdarkly-ai-litellm-agents-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
* **client:** capture conversation content by default
([#213](#213))

### Features

* **client:** add result variant and value to the feature_flag span
event ([#240](#240))
([040d487](040d487))
* **client:** capture conversation content by default
([#213](#213))
([77a7cf5](77a7cf5))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3573:** never send a NativeTool to the model as a function tool
([#248](#248))
([73caeae](73caeae))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))
</details>

<details><summary>launchdarkly-ai-openai-messages: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-openai-messages-0.3.0...launchdarkly-ai-openai-messages-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
* **client:** capture conversation content by default
([#213](#213))

### Features

* **client:** add result variant and value to the feature_flag span
event ([#240](#240))
([040d487](040d487))
* **client:** capture conversation content by default
([#213](#213))
([77a7cf5](77a7cf5))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3573:** never send a NativeTool to the model as a function tool
([#248](#248))
([73caeae](73caeae))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))
* **AIC-3726:** realign three usage IDs with the 1.0 factory names
([#252](#252))
([1c4d904](1c4d904))
</details>

<details><summary>launchdarkly-ai-vercel-messages: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-vercel-messages-0.2.0...launchdarkly-ai-vercel-messages-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))

### Features

* **client:** add result variant and value to the feature_flag span
event ([#240](#240))
([040d487](040d487))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3573:** never send a NativeTool to the model as a function tool
([#248](#248))
([73caeae](73caeae))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))
</details>

<details><summary>launchdarkly-ai-litellm-messages: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-litellm-messages-0.2.0...launchdarkly-ai-litellm-messages-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
* **client:** capture conversation content by default
([#213](#213))

### Features

* **client:** add result variant and value to the feature_flag span
event ([#240](#240))
([040d487](040d487))
* **client:** capture conversation content by default
([#213](#213))
([77a7cf5](77a7cf5))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3573:** never send a NativeTool to the model as a function tool
([#248](#248))
([73caeae](73caeae))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))
</details>

<details><summary>launchdarkly-ai-langchain-agents: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-langchain-agents-0.3.0...launchdarkly-ai-langchain-agents-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **evaluations:** route generation and judges to a scoped wildcard
handler only for its named providers, and accept two scoped wildcards of
one mode when their lists differ
* **AIC-3439:** let wildcard handlers accept only named providers
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
* **client:** capture conversation content by default
([#213](#213))

### Features

* **AIC-3439:** let wildcard handlers accept only named providers
([454a4de](454a4de))
* **client:** add result variant and value to the feature_flag span
event ([#240](#240))
([040d487](040d487))
* **client:** capture conversation content by default
([#213](#213))
([77a7cf5](77a7cf5))
* **evaluations:** route generation and judges to a scoped wildcard
handler only for its named providers, and accept two scoped wildcards of
one mode when their lists differ
([454a4de](454a4de))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3573:** never send a NativeTool to the model as a function tool
([#248](#248))
([73caeae](73caeae))
* **AIC-3590:** build Python langchain-agents with
langchain.agents.create_agent
([#175](#175))
([8218626](8218626))
* **AIC-3592:** make native LangGraph hand-offs complete
([#246](#246))
([bfb738f](bfb738f))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))
</details>

<details><summary>launchdarkly-ai-langchain-messages: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-langchain-messages-0.3.0...launchdarkly-ai-langchain-messages-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **evaluations:** route generation and judges to a scoped wildcard
handler only for its named providers, and accept two scoped wildcards of
one mode when their lists differ
* **AIC-3439:** let wildcard handlers accept only named providers
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
* **client:** capture conversation content by default
([#213](#213))

### Features

* **AIC-3439:** let wildcard handlers accept only named providers
([454a4de](454a4de))
* **client:** add result variant and value to the feature_flag span
event ([#240](#240))
([040d487](040d487))
* **client:** capture conversation content by default
([#213](#213))
([77a7cf5](77a7cf5))
* **evaluations:** route generation and judges to a scoped wildcard
handler only for its named providers, and accept two scoped wildcards of
one mode when their lists differ
([454a4de](454a4de))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3573:** never send a NativeTool to the model as a function tool
([#248](#248))
([73caeae](73caeae))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))
* **AIC-3726:** realign three usage IDs with the 1.0 factory names
([#252](#252))
([1c4d904](1c4d904))
</details>

<details><summary>launchdarkly-ai-opencode-agents: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-opencode-agents-0.3.0...launchdarkly-ai-opencode-agents-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))

### Features

* **client:** add result variant and value to the feature_flag span
event ([#240](#240))
([040d487](040d487))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))
</details>

<details><summary>launchdarkly-ai-typesafe: 1.0.0</summary>

##
[1.0.0](launchdarkly-ai-typesafe-0.2.0...launchdarkly-ai-typesafe-1.0.0)
(2026-10-30)


### ⚠ BREAKING CHANGES

* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
* **client:** capture conversation content by default
([#213](#213))

### Features

* **client:** add result variant and value to the feature_flag span
event ([#240](#240))
([040d487](040d487))
* **client:** capture conversation content by default
([#213](#213))
([77a7cf5](77a7cf5))


### Bug Fixes

* **AIC-3568:** give the barrel core's __all__, and drop private-spec
references
([#234](#234))
([88e57e6](88e57e6))
* **AIC-3598:** make Python packaging and public types GA-ready
([#218](#218))
([5860b40](5860b40))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> This is a **Release Please** cut that moves every monorepo package to
**1.0.0** and records the GA release in changelogs. It updates
`.release-please-manifest.json`, each package’s `pyproject.toml` /
`__version__`, and prepends **1.0.0** notes to every `CHANGELOG.md`.
> 
> Handler and barrel packages now require
**`launchdarkly-ai-server>=1.0.0,<2`**, so installs must upgrade the
core client in lockstep with adapters.
> 
> There is **no new application logic** in this diff—only release
metadata. The changelog text still matters for reviewers: it summarizes
breaking work already on `main`, including **Evaluations from Code
promoted to core**, **conversation content captured by default**,
**`init_client` failing when LD init fails**, telemetry
**`launchdarkly.project_id`**, renames like **`LaunchDarklySkillStore`**
and **`create_openai_agents_handler`**, tightened **wildcard handler
routing**, and **GA packaging / public `__all__`** cleanup.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
2e904f7. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@donei003 donei003 donei003 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@aknight-ld @donei003