[hsadhvani]

Why

Shadow requests let a team try a candidate variation on real traffic without serving it and compare it on the normal AI metrics. This replaces the closed #241, which shipped it as experimental behind a new core invocation hook. We are not shipping new features as experimental, and a judges-style core design is easier to read.

Summary

Adds a core shadows.py module that config().invoke() and .stream() call directly, the same way they call judges. config() evaluates once with variation_detail, picks the shadows for the matched rule or fallthrough, and runs the sampled ones in the background with shadow_timeout_s (default 30). skip_shadows=True returns them as picklable ShadowTasks for run_shadow(task, handlers). Shadow events reuse A's runId and add isShadow, primaryVariationKey, shadowRequestId, ownerKind and ownerId. shutdown() cancels running shadows.

Testing approaches

• Added tests for slot selection by evaluation reason, expired entries, missing or disabled variations, and shadows with no handler.
• Added tests for sticky sampling, the 20% rate over 10,000 keys, the rate edges and a context with no key.
• Added tests that a failing or timed out shadow leaves the caller's response and events unchanged, and that a timeout is tracked once.
• Added tests for deferred tasks on invoke and the stream done event, pickling, and matching events between background and deferred runs.
• Added tests for the shadow event fields, the root span attributes, and that template variables cannot set span attributes.
• Updated the public surface snapshot and the API report.

Feature flags

N/A

---

Note

Overview
Adds shadow requests to config().invoke() and .stream(): after the primary variation is chosen, the SDK can run other pinned variations on the same input in the background, emit normal $ld:ai:* metrics with isShadow: true, and never change the caller’s response, latency, or primary events.

Evaluation and wiring. extract_variation(..., with_reason=True) uses variation_detail when available so shadows attach to the matched rule or fallthrough slot from shadowConfiguration. Primary and shadow runs share one runId. execute_and_track / streaming gain optional run_id, extra_track_data, and handler span_attributes (via __ld_span) for shadow OTel tags.

API surface. config() adds skip_shadows and shadow_timeout_s (default 30s). With skip_shadows=True, responses expose picklable ShadowTask lists (invoke) or on the stream done event; workers run them with run_shadow, mirroring deferred judges. New types ShadowTask / ShadowResult are exported; README and the API report are updated. shutdown() cancels in-flight shadow tasks.

Shadow execution rules. Sampled shadows use a stripped config (no tools, judges, or nested shadow config), no tool calls, timeout-wrapped handlers, and errors/timeouts recorded on ShadowResult without affecting the primary path. Extensive test_shadows.py plus test mocks that drop variation_detail when simulating older clients.

^{Reviewed by Cursor Bugbot for commit 378a915. Bugbot is set up for automated code reviews on this repo. Configure here.}