[hsadhvani]

Why

LaunchDarkly can now attach shadow requests to an AI Config's targeting rules and fallthrough, and delivers them inside the served variation as shadowConfiguration (AIC-3429). The SDK needs to run them: while serving variation A, also run variation B in the background on the same input and report B on the normal AI metrics, so customers can compare models on real traffic. Draft until the Shadow Testing UI ships.

Summary

Shadow requests run from core, the same way judges do. A new shadows.py picks the shadow requests for the matched rule or fallthrough, skips ended ones, samples sticky per context, strips tools and judges, and runs them in the background with a 30 second timeout. config().invoke() and .stream() call it directly.

• config(skip_shadows=True) returns shadow_tasks on the response instead of running them, and run_shadow(task, handlers) runs one later, mirroring skip_judges and run_judge.
• config(shadow_timeout_s=...) sets the timeout.
• config() evaluates with variation_detail() when the client has it, to read the evaluation reason. Graphs, judges and evaluations still call variation().
• Shadow events add isShadow, primaryVariationKey, shadowRequestId, ownerKind, ownerId and the real run's runId. The shadow's root span carries launchdarkly.is_shadow, the shadow request id and the primary variation key.
• execute_and_track / execute_and_stream take optional run_id, extra_track_data and span_attributes. This shows in the root API report.

Nothing to turn on in the SDK: LaunchDarkly only sends shadow requests while enable-shadow-requests is on and someone has set one up.

Testing approaches

• Added tests for selecting shadows by reason, end time, missing variations and missing handlers
• Added tests for sticky sampling, its rate and parity with the TypeScript bucket
• Added tests that a failing or timed out shadow never reaches the caller and is tracked once
• Added tests for skip_shadows, picklable tasks and matching events between background and deferred runs
• Added tests for shadow event fields, unchanged real events and shadow span attributes

Feature flags

N/A. LaunchDarkly only sends shadow requests while enable-shadow-requests is on.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 4c5efa1. Configure here.}

[cursor]

Shadow setup errors fail live calls

Medium Severity

Building and starting shadows now runs inline on invoke() and stream() with no outer guard. The old invocation-hook runner swallowed hook failures so the real call still ran. build_shadow_tasks only catches _entries_for, so a later parse, dict(context), or start_shadows error now fails the customer's live request even though shadow work is supposed to be best-effort.

Additional Locations (2)

• packages/client/src/launchdarkly_ai_server/client.py#L287-L301
• packages/client/src/launchdarkly_ai_server/shadows.py#L68-L116

 

^{Reviewed by Cursor Bugbot for commit 4c5efa1. Configure here.}