test(AIC-3716): run the OpenAI native graph with a tool on the real SDK by jeffdupont · Pull Request #196 · launchdarkly/js-ai-sdk

jeffdupont · 2026-10-29T20:30:36Z

Parity test for AIC-3716. Python fix: launchdarkly/python-ai-sdk#251.

Why

The Python OpenAI native graph crashed on every node with a tool, because it called agents.tool, which is a submodule. Its tests replaced the whole SDK with a mock, so CI never saw the crash. native-graph.test.ts here also mocks all of @openai/agents, so it could not catch the same class of bug. This PR adds the real-import test that AIC-3716 asks for in each SDK.

Change

Test only. The new src/__tests__/native-graph.real-sdk.test.ts imports the real @openai/agents (0.11.6) and fakes only the model, via setDefaultModelProvider. It runs a node with a tool through toOpenAIAgents. It is a separate file because vi.mock applies to a whole file.

The test also calls setTracingDisabled(true), so no trace of the run can be exported to OpenAI. It is a guard only: with OPENAI_API_KEY=sk-fake I saw no export even without it.

It asserts:

the tool call reaches the handler with the parsed arguments
the run returns the model's final answer
the model receives the tool with the config's name, description and schema unchanged, and with strict: false

The JS graph is not affected: tool() is a function and the test passes as is. I checked that the test can fail. With strict: true it fails on the schema assertion. With node tools dropped, it fails with Tool lookup not found in agent root.

Spec

No change needed. TESTING.md §2.2 "Tool forwarding" already covers this, and JS meets it.

Checks

code:check, typecheck, api:check and test all pass.

🤖 Generated with Claude Code

Note

Overview
Adds native-graph.real-sdk.test.ts, a parity guard for AIC-3716: it runs toOpenAIAgents against the real @openai/agents package (only the model is faked via setDefaultModelProvider), unlike native-graph.test.ts which mocks the entire SDK and would miss tool-builder bugs like the Python crash.

The test drives a single-node graph with a lookup tool through a two-turn fake model (tool call, then final message). It asserts the handler receives parsed args, the run completes with the model answer, and the model request includes the tool with the graph’s name, description, schema, and strict: false. setTracingDisabled(true) keeps test runs from exporting traces regardless of API key env.

^{Reviewed by Cursor Bugbot for commit a9c1db9. Bugbot is set up for automated code reviews on this repo. Configure here.}

native-graph.test.ts mocks all of @openai/agents, so it cannot catch a tool builder that the real SDK rejects. That is how the Python adapter shipped one that crashed on every node with a tool (AIC-3716). Add a test that imports the real package, fakes only the model, and runs a node with a tool: the call reaches the handler, the run completes, and the model receives the config schema unchanged with strict: false. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…#251) Fixes [AIC-3716](https://launchdarkly.atlassian.net/browse/AIC-3716), a 1.0 blocker. JS parity test: launchdarkly/js-ai-sdk#196. ## Problem `to_openai_agents` crashes on every graph node that has a tool: ``` File ".../launchdarkly_ai_openai_agents/native_graph.py", line 92, in _build_node_tools TypeError: 'module' object is not callable ``` `_build_node_tools` called `agents.tool`, which is a submodule in openai-agents, not a function. This happens in 0.17.7 (locked) and in 0.24.0 (latest). Nodes without tools were unaffected. CI was green because every test replaced the whole `agents` module with a `MagicMock`, and a mock accepts any call. ## Change - Node tools are built with `agents.FunctionTool`, the class `handler.py` already uses. `agents.function_tool` is not usable here because it reads the schema from the Python function signature and has no `params_json_schema` argument. - The tool name, description, schema and handler wiring are unchanged. `on_invoke_tool` decodes the JSON arguments the same way `handler.py` does. A missing handler or a `NativeTool` still returns `""`, and sync or async handlers still go through `tool_result_string`. - `strict_json_schema=False`, so the config schema reaches the model as given. This matches the TypeScript graph's `strict: false`. Strict mode would make every property required, add `additionalProperties: false`, and reject `additionalProperties: true`. - Handler errors propagate, matching the TypeScript graph's `errorFunction: null`. A directly built `FunctionTool` does not wrap `on_invoke_tool`. - `openai-agents>=0.17,<0.25`. openai-agents makes breaking changes in minor (`0.Y`) releases (its `docs/release.md`), and 0.24.0 is the latest, so the bound stops at 0.25. `uv lock --check` passes; the lockfile changes only the specifier. ## Tests - `TestBuildNodeToolsSyncHandlers` now builds tools with the real `FunctionTool` instead of patching `importlib.import_module`. - New `TestNativeGraphRunsANodeWithAToolOnTheRealSdk` runs `to_openai_agents` with the real `agents` package and fakes only the model (`MultiProvider.get_model`). It asserts that the tool call reaches the handler with the parsed arguments, that the run returns the model's final answer, and that the model receives a `FunctionTool` with the config's name, description and schema unchanged. The schema has an optional property, so a strict-mode rewrite would fail the test. - **On `main` 8218626, all 4 of these tests fail** with the ticket's `TypeError`. With this change they pass on 0.17.7, and the native graph suite passes on 0.24.0 (`uv run --with openai-agents==0.24.0`). - The stale `agents_mock.tool` stubs are removed. - The real-SDK test turns off the SDK's trace export for its run, so a developer with `OPENAI_API_KEY` set sends nothing to OpenAI. I checked this with `OPENAI_API_KEY=sk-fake`: without the patch the run logs a 401 from the trace exporter, and with it there is no request. ## Spec No change needed. TESTING.md §2.2 "Tool forwarding" already requires node tools "in that provider's native tool format". Python broke that rule, and this PR fixes it. ## Checks `make lint format-check typecheck api-report` pass and `uv lock --check` passes. `make test`: 3714 passed, 18 skipped, and 2 failed in `packages/client/tests/test_lifecycle.py::TestInitClientSDKKeyPath`. Those 2 fail the same way locally on clean `main` 8218626, pass when that file runs alone, and this PR does not touch the client package. ## Notes - **Known difference from JS, kept on purpose:** tool arguments that are not valid JSON become `{}` and the handler still runs, as in Python's `handler.py`. JS `tool()` raises `InvalidToolInputError` and fails the run. Changing this would make Python's handler and native graph behave differently, so it is left for a follow-up that aligns both Python paths with JS. - #248 (AIC-3573) edits the same loop in `_build_node_tools`, adding an `is_function_tool` skip. Whichever merges second gets a small conflict. - On openai-agents 0.24.0, `test_parameter_forwarding.py::test_every_field_is_classified_exactly_once` fails because `ModelSettings` gained `preserve_raw_usage`, `prompt_cache_options` and `timeout`. This is not caused by this PR, and CI is locked to 0.17.7. It needs a separate ticket if the lock moves. 🤖 Generated with [Claude Code](https://claude.com/claude-code) [AIC-3716]: https://launchdarkly.atlassian.net/browse/AIC-3716?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ  --- > [!NOTE] > **Overview** > Fixes a **1.0 blocker** where `to_openai_agents` crashed on any graph node that declared tools: `_build_node_tools` called `agents.tool`, which is a **submodule**, not a decorator—raising `TypeError: 'module' object is not callable`. > > Node tools are now built with **`agents.FunctionTool`**, matching the single-node handler. Tool callbacks use **`on_invoke_tool`** with JSON argument decoding, default empty-object schemas when missing, and **`strict_json_schema=False`** so LaunchDarkly config schemas reach the model unchanged (TypeScript `strict: false` parity). > > The **`openai-agents`** dependency is capped at **`>=0.17,<0.25`** because the SDK ships breaking changes in minor releases. > > Tests stop mocking a fake `agents.tool` and instead exercise the real **`FunctionTool`**; a new integration test runs the native graph through the real Agents SDK with only the model faked, asserting tool calls reach handlers and schemas are not strict-rewritten. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1969deb. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>  --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Steps 1 and 3 of [AIC-3673](https://launchdarkly.atlassian.net/browse/AIC-3673). The Python half is launchdarkly/python-ai-sdk#242. ## What this does - **`release-as: 1.0.0`** on all 14 packages in `release-please-config.json`. The `node-workspace` plugin updates the internal dependency ranges. - **Publish jobs for `litellm-agents` and `litellm-messages`.** They had none, so `@launchdarkly/ai-litellm-*-0.2.0` was tagged but never published. Both are also added to the manual-publish choices. `changelog-sections` is left alone: #181 (merged) removed the evaluations Experimental rows when it promoted Evaluations from Code to core. ## ⚠️ npm setup: trusted publishers still to confirm for 5 of 14 packages The 26 Oct release run ([40393827352](https://github.com/launchdarkly/js-ai-sdk/actions/runs/40393827352)) failed 3 publish jobs, and the litellm packages have never had a job: | Package | On npm | Failure | |---|---|---| | `@launchdarkly/ai-typesafe` | 0.0.0 placeholder | `E404 … PUT @launchdarkly/ai-typesafe`: the package doesn't exist, and OIDC can't create it | | `@launchdarkly/ai-vercel-agents` | 0.0.0 placeholder | `E403 OIDC permission denied` | | `@launchdarkly/ai-vercel-messages` | 0.0.0 placeholder | `E403 OIDC permission denied` | | `@launchdarkly/ai-litellm-agents` | 0.0.0 placeholder | no publish job (added here) | | `@launchdarkly/ai-litellm-messages` | 0.0.0 placeholder | no publish job (added here) | Before the release PR merges, an npm org admin needs to: 1. ~~Publish a 0.0.0 placeholder for `ai-typesafe`, `ai-litellm-agents` and `ai-litellm-messages`.~~ Done: all three are on npm at 0.0.0 (checked 29 Oct). 2. Add a trusted publisher (`launchdarkly/js-ai-sdk`, workflow `release-please.yml`) to all five. Otherwise 1.0.0 ships 9 of 14 packages. ## Merge this last Merge this only after the items below are done. Anything merged before the release PR (#171) merges ships in 1.0.0, and a `!` change merged after 1.0 forces 2.0. - [x] #181 feat(evaluations)!: promote Evaluations from Code to core. After it, `@launchdarkly/ai-server/experimental` exports nothing. - [x] #180 fix(SEC-9507)!: dispatch tools only from the active variation's config.tools - [x] #183 fix(claude-agents)!: honour max_tokens and disable thinking by default - [x] #111 feat(AIC-3439)!: wildcard handlers accept only named providers (merged 29 Oct) - [x] npm placeholders (above) - [ ] Trusted publishers on all five packages (above; needs an npm org admin to confirm) - [ ] #192 fix(AIC-3592): make native LangGraph hand-offs complete (ready for review; spec launchdarkly/ai-sdks-monorepo#118) - [ ] #194 fix(AIC-3573): never send a NativeTool to the model as a function tool (ready for review; spec launchdarkly/ai-sdks-monorepo#116) - [ ] #190 feat(client)!: capture conversation content by default and add configure() (approved; breaking, so it must land before #171) - [ ] #196 test(AIC-3716): run the OpenAI native graph with a tool on the real SDK (test only; optional for the release) Not in 1.0: #187 (Shadow Requests, closed). 1.0 ships with nothing experimental. ## After merging 1. Confirm the regenerated #171 lists 1.0.0 for all 14 packages and its CI is green, then merge it. 2. Install each package from npm into an empty project and import its entry points, in both ESM and CJS (AIC-3673 step 5). 3. Open a PR removing `release-as` straight away. ## Checks - The workflow YAML parses: 14 per-package publish jobs plus release-please and the canary. The new jobs are copies of `release-typesafe`. - No local build was run: this is config only, made through the GitHub API. 🤖 Generated with [Claude Code](https://claude.com/claude-code) [AIC-3673]: https://launchdarkly.atlassian.net/browse/AIC-3673?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ  --- > [!NOTE] > **Overview** > Prepares the monorepo for a coordinated **1.0.0** npm release by setting **`release-as: "1.0.0"`** on all 14 workspaces in `release-please-config.json` (the `node-workspace` plugin will align internal dependency ranges when the release PR lands). > > The **`release-please`** workflow now fully covers **`litellm-agents`** and **`litellm-messages`**: they appear in manual `workflow_dispatch` package choices, expose release-please job outputs, and get dedicated **`release-litellm-*`** publish jobs (build, attest, npm publish) matching the other packages—closing the gap where LiteLLM tags could be created without a publish step. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit af882d4. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>  Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: andrewklatzke <aklatzke@launchdarkly.com>

jeffdupont mentioned this pull request Oct 29, 2026

fix(AIC-3716): build OpenAI native graph node tools with FunctionTool launchdarkly/python-ai-sdk#251

Merged

test(AIC-3716): disable tracing in the real-SDK graph test

a9c1db9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

jeffdupont marked this pull request as ready for review October 29, 2026 20:43

jeffdupont mentioned this pull request Oct 29, 2026

chore(AIC-3673): release every JS package as 1.0.0 #188

Merged

10 tasks done

aknight-ld approved these changes Oct 30, 2026

View reviewed changes

jeffdupont merged commit a8ab660 into main Oct 30, 2026
15 checks passed

jeffdupont deleted the jdupont/aic-3716-openai-native-graph-tools branch October 30, 2026 14:45

Navigation Menu

Repository navigation

test(AIC-3716): run the OpenAI native graph with a tool on the real SDK - #196

test(AIC-3716): run the OpenAI native graph with a tool on the real SDK#196
jeffdupont merged 2 commits into
mainfrom
jdupont/aic-3716-openai-native-graph-tools

jeffdupont commented Oct 29, 2026 •

edited by cursor Bot

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jeffdupont commented Oct 29, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Change

Spec

Checks

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jeffdupont commented Oct 29, 2026 •

edited by cursor Bot

Loading