Skip to content

feat(shadow-requests): run Shadow Requests from the experimental entry point - #187

Closed
hsadhvani wants to merge 2 commits into
mainfrom
aic-3514-shadow-requests
Closed

hsadhvani wants to merge 2 commits into
mainfrom
aic-3514-shadow-requests

Conversation

@hsadhvani

Copy link
Copy Markdown

Why

LaunchDarkly can now attach Shadow Requests to an AI Config's targeting rules and fallthrough, and delivers them inside the served variation as shadowConfiguration (AIC-3429). The SDK needs to run them: while serving variation A, also run variation B in the background on the same input and report B on the normal AI metrics, so customers can compare models on real traffic. This is the TypeScript counterpart of launchdarkly/python-ai-sdk#241, experimental and opt in until the UI ships.

Summary

New exports from @launchdarkly/ai-server/experimental (and so @launchdarkly/ai-node/experimental): enableShadowRequests({ timeoutMs, onTask }), disableShadowRequests(), runShadow(), and the types ShadowTask, ShadowResult and EnableShadowRequestsOptions. Selection, sampling (same bucket as Python), stripping and event fields match the Python SDK. ShadowTask is plain data, so it survives structuredClone and JSON for apps that run shadows on their own queue.

Core changes, none naming the feature (spec §0.3):

  • A generic invocation hook registry (invocation-hooks.ts), run by config().invoke() and .stream() before the handler.
  • extractVariation calls variationDetail() only while a hook is registered and the client has it, so apps that never enable this keep calling variation(). LDClientInterface gains an optional variationDetail.
  • executeAndTrack / executeAndStream / makeRunTrackData take optional runId, extraTrackData and spanAttributes. This shows in the root API report.

Spec: TESTING.md §9 and A.22 in ai-sdks-monorepo (launchdarkly/ai-sdks-monorepo#114). Draft until the Shadow Testing UI is ready.

BEGIN_COMMIT_OVERRIDE
feat(shadow-requests): run Shadow Requests from the experimental entry point
feat(client): add invocation hooks and runId, extraTrackData and spanAttributes options to executeAndTrack
END_COMMIT_OVERRIDE

Testing approaches

  • Added tests for selecting shadows by reason, end time, missing variations and missing handlers
  • Added tests for sticky sampling, its rate and parity with the Python bucket
  • Added tests that a failing or timed out shadow never reaches the caller and records an error
  • Added tests for deferred mode, serializable tasks and matching events between modes
  • Added tests for shadow event fields, unchanged real events and shadow span attributes
  • Added tests for the invocation hook registry and the experimental export list

Feature flags

N/A. Off unless the app calls enableShadowRequests(); LaunchDarkly only sends shadows while enable-shadow-requests is on.

@hsadhvani hsadhvani closed this Oct 29, 2026
jeffdupont added a commit that referenced this pull request Oct 30, 2026
Steps 1 and 3 of
[AIC-3673](https://launchdarkly.atlassian.net/browse/AIC-3673). The
Python half is launchdarkly/python-ai-sdk#242.

## What this does

- **`release-as: 1.0.0`** on all 14 packages in
`release-please-config.json`. The `node-workspace` plugin updates the
internal dependency ranges.
- **Publish jobs for `litellm-agents` and `litellm-messages`.** They had
none, so `@launchdarkly/ai-litellm-*-0.2.0` was tagged but never
published. Both are also added to the manual-publish choices.

`changelog-sections` is left alone: #181 (merged) removed the
evaluations Experimental rows when it promoted Evaluations from Code to
core.

## ⚠️ npm setup: trusted publishers still to confirm for 5 of 14
packages

The 26 Oct release run
([40393827352](https://github.com/launchdarkly/js-ai-sdk/actions/runs/40393827352))
failed 3 publish jobs, and the litellm packages have never had a job:

| Package | On npm | Failure |
|---|---|---|
| `@launchdarkly/ai-typesafe` | 0.0.0 placeholder | `E404 … PUT
@launchdarkly/ai-typesafe`: the package doesn't exist, and OIDC can't
create it |
| `@launchdarkly/ai-vercel-agents` | 0.0.0 placeholder | `E403 OIDC
permission denied` |
| `@launchdarkly/ai-vercel-messages` | 0.0.0 placeholder | `E403 OIDC
permission denied` |
| `@launchdarkly/ai-litellm-agents` | 0.0.0 placeholder | no publish job
(added here) |
| `@launchdarkly/ai-litellm-messages` | 0.0.0 placeholder | no publish
job (added here) |

Before the release PR merges, an npm org admin needs to:
1. ~~Publish a 0.0.0 placeholder for `ai-typesafe`, `ai-litellm-agents`
and `ai-litellm-messages`.~~ Done: all three are on npm at 0.0.0
(checked 29 Oct).
2. Add a trusted publisher (`launchdarkly/js-ai-sdk`, workflow
`release-please.yml`) to all five.

Otherwise 1.0.0 ships 9 of 14 packages.

## Merge this last

Merge this only after the items below are done. Anything merged before
the release PR (#171) merges ships in 1.0.0, and a `!` change merged
after 1.0 forces 2.0.

- [x] #181 feat(evaluations)!: promote Evaluations from Code to core.
After it, `@launchdarkly/ai-server/experimental` exports nothing.
- [x] #180 fix(SEC-9507)!: dispatch tools only from the active
variation's config.tools
- [x] #183 fix(claude-agents)!: honour max_tokens and disable thinking
by default
- [x] #111 feat(AIC-3439)!: wildcard handlers accept only named
providers (merged 29 Oct)
- [x] npm placeholders (above)
- [ ] Trusted publishers on all five packages (above; needs an npm org
admin to confirm)
- [ ] #192 fix(AIC-3592): make native LangGraph hand-offs complete
(ready for review; spec launchdarkly/ai-sdks-monorepo#118)
- [ ] #194 fix(AIC-3573): never send a NativeTool to the model as a
function tool (ready for review; spec launchdarkly/ai-sdks-monorepo#116)
- [ ] #190 feat(client)!: capture conversation content by default and
add configure() (approved; breaking, so it must land before #171)
- [ ] #196 test(AIC-3716): run the OpenAI native graph with a tool on
the real SDK (test only; optional for the release)

Not in 1.0: #187 (Shadow Requests, closed). 1.0 ships with nothing
experimental.

## After merging

1. Confirm the regenerated #171 lists 1.0.0 for all 14 packages and its
CI is green, then merge it.
2. Install each package from npm into an empty project and import its
entry points, in both ESM and CJS (AIC-3673 step 5).
3. Open a PR removing `release-as` straight away.

## Checks

- The workflow YAML parses: 14 per-package publish jobs plus
release-please and the canary. The new jobs are copies of
`release-typesafe`.
- No local build was run: this is config only, made through the GitHub
API.

🤖 Generated with [Claude Code](https://claude.com/claude-code)


[AIC-3673]:
https://launchdarkly.atlassian.net/browse/AIC-3673?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Prepares the monorepo for a coordinated **1.0.0** npm release by
setting **`release-as: "1.0.0"`** on all 14 workspaces in
`release-please-config.json` (the `node-workspace` plugin will align
internal dependency ranges when the release PR lands).
> 
> The **`release-please`** workflow now fully covers
**`litellm-agents`** and **`litellm-messages`**: they appear in manual
`workflow_dispatch` package choices, expose release-please job outputs,
and get dedicated **`release-litellm-*`** publish jobs (build, attest,
npm publish) matching the other packages—closing the gap where LiteLLM
tags could be created without a publish step.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
af882d4. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: andrewklatzke <aklatzke@launchdarkly.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant