feat(AIC-3439)!: let wildcard handlers accept only named providers

AGENTS.md

@@ -384,7 +384,7 @@ Returns a `ConfigInstance` with:
 **Behavior when `.invoke()` is called:**
 
 1. Fetches and validates the `AiConfigRep` variation from LaunchDarkly using `key` and the supplied `context`. Raises if the variation is disabled or invalid.
-2. Selects the handler by matching on `[config.provider.name, normalized mode]`. Selection priority: (a) exact provider match, (b) wildcard `['*', mode]` fallback for multi-provider adapters (e.g. LangChain). Raises if no matching handler is found.
+2. Selects the handler by matching on `[config.provider.name, normalized mode]`. Selection priority: (a) exact provider match, ignoring any `providers` list; (b) a wildcard `['*', mode]` whose `providers` list contains the name, shortest list first and earlier registration on a tie; (c) an unscoped wildcard `['*', mode]`. Raises if no matching handler is found.
 3. Invokes the selected handler with the config, user input, tool handlers, variables, and history. The `context` passed to `.invoke()` is automatically merged into `variables` under the key `ldContext`, so templates can reference `{{ldContext.key}}`, `{{ldContext.email}}`, etc. If `history` is provided, it is passed to the handler as the 5th positional argument — messages-mode handlers splice it into the messages array; agent-mode handlers append it to the system prompt.
 4. Emits LaunchDarkly telemetry events: duration (`$ld:ai:duration:total`), outcome (`$ld:ai:generation:success` / `$ld:ai:generation:error`), and token counts (`$ld:ai:tokens:*`).
 5. If `judgeConfiguration` is present:
@@ -419,7 +419,7 @@ registry = Registry(
 )
 ```
 
-`.register(handlers=[], tools={})` can be called multiple times to add more handlers or tools. Duplicate `provides_for` keys or tool names produce a warning and the last registration wins.
+`.register(handlers=[], tools={})` can be called multiple times to add more handlers or tools. Handler identity is `provides_for` plus the provider set: two wildcards of the same mode with different `providers` lists both stay registered, while the same set (order-independent; repeated names collapse) or two unscoped wildcards warn and the last registration wins. Duplicate tool names do the same.
 
 `global_registry` is a pre-constructed singleton `Registry` instance.
 
@@ -470,7 +470,7 @@ async def handler(
 handler.provides_for = [provider_name: str, mode: Literal["agent", "messages"]]
 ```
 
-The `provides_for` list is how `config()` routes to the correct handler at runtime. The mode element must exactly match the normalized `meta.mode`. The provider element must either exactly match `config.provider.name` **or** be the wildcard `'*'`. A wildcard handler is chosen only when no handler with an exact provider name matches — it acts as a fallback for multi-provider adapters like LangChain. **Always attach `provides_for` using `create_handler` rather than direct attribute assignment.**
+The `provides_for` list is how `config()` routes to the correct handler at runtime. The mode element must exactly match the normalized `meta.mode`. The provider element must either exactly match `config.provider.name` **or** be the wildcard `'*'`. A wildcard handler is chosen only when no handler with an exact provider name matches — it acts as a fallback for multi-provider adapters like LangChain. Pass `providers=["Bedrock"]` to `create_handler` to limit which `config.provider.name` values that wildcard accepts; names must match exactly, and an empty list or a blank name is rejected. Omit `providers` to accept every provider. **Always attach `provides_for` using `create_handler` rather than direct attribute assignment.**
 
 ### Factory Function
 

packages/client/README.md

@@ -197,7 +197,7 @@ Two limits keep a trajectory from spending the judge's context window: at most 5
 
 A tool result is now judge-prompt input. It stays literal for the same reason the generated output does: the judge config is handed to the handler unrendered and the handler makes exactly one template pass, so a `{{...}}` sequence coming back from a tool is never expanded into the judge prompt.
 
-**Judges are independent AI Configs, so handlers are routed per judge.** A judge may resolve to a different provider or mode than `generation`. Each judge selects its handler from `handler` or `handlers` by its provider and mode, with the same rule as generation. There is no mode fallback: a messages-mode judge never runs on an agent handler. A judge's prompt must also match its handler's mode. Judges get no tools: the harness passes an empty tool map and removes `tools` from the judge config.
+**Judges are independent AI Configs, so handlers are routed per judge.** A judge may resolve to a different provider or mode than `generation`. Each judge selects its handler from `handler` or `handlers` by its provider and mode, with the same rule as generation. There is no mode fallback: a messages-mode judge never runs on an agent handler. A judge's prompt must also match its handler's mode. Judges get no tools: the harness passes an empty tool map and removes `tools` from the judge config. A wildcard's `providers` list, when set, accepts only those names: an exact provider match wins, then the shortest list that contains the name, then an unscoped `*`.
 
 Judges are resolved through flag delivery, and handlers are matched to them, **before** any evaluation records are created — a missing judge, a judge that no handler covers, or a judge whose prompt does not match its handler fails the run up front rather than after the generation spend. One error lists every judge with a problem. After that point a criterion failure never aborts the run: an unparseable judge response, an out-of-range score, and a raising handler or scorer each become a per-criterion `ERROR` event carrying a top-level `errorMessage` that describes the failure. A row whose generation errored is not scored at all: no judge or scorer runs for it and no criterion event is sent, because its generation `ERROR` event already records the row as failed. Event *delivery* is different: the backend needs one result per `(row, criterion)` to finish row accounting, so if tracking a criterion event fails, every remaining result is still attempted and flushed and then `run()` raises — rather than polling to its timeout with the cause hidden.
 

packages/client/api-report/launchdarkly_ai_server.txt

@@ -24,7 +24,7 @@ ConversationIdSpanProcessor(_SpanProcessorBase)  [class]
   on_end(span: Any) -> None
   on_start(span: Span, parent_context: otel_context.Context | None = None) -> None
   shutdown() -> None
-create_handler(provides_for: tuple[str, Literal['agent', 'messages']], fn: _HandlerFn, stream_fn: _StreamFn | None = None, capture_content: bool | None = None) -> ProviderHandler
+create_handler(provides_for: tuple[str, Literal['agent', 'messages']], fn: _HandlerFn, stream_fn: _StreamFn | None = None, capture_content: bool | None = None, *, providers: Sequence[str] | None = None) -> ProviderHandler
 create_run_usage() -> RunUsage
 Criterion = Judge | Scorer  [type]
 DatasetRow  [class, @dataclass]
@@ -250,10 +250,11 @@ ProviderGraphResponse  [class, @dataclass]
   response: str
   usage: UsageDict
 ProviderHandler  [class]
-  __init__(fn: _HandlerFn, provides_for: tuple[str, Literal['agent', 'messages']] | None = None, stream_fn: _StreamFn | None = None, capture_content: bool | None = None)
+  __init__(fn: _HandlerFn, provides_for: tuple[str, Literal['agent', 'messages']] | None = None, stream_fn: _StreamFn | None = None, capture_content: bool | None = None, *, providers: Sequence[str] | None = None)
   __call__(config: AiConfigRep, user_input: str | None = None, tool_handlers: Mapping[str, Callable[..., Any] | NativeTool] | None = None, variables: dict[str, Any] | None = None, history: list[dict[str, Any]] | None = None) -> HandlerResult  [async]
   capture_content: bool  [property]
   has_stream: bool  [property]
+  providers: tuple[str, ...] | None = normalize_providers(providers)
   provides_for: tuple[str, Literal['agent', 'messages']] | None = provides_for
   stream(config: AiConfigRep, user_input: str | None = None, tool_handlers: Mapping[str, Callable[..., Any] | NativeTool] | None = None, variables: dict[str, Any] | None = None, history: list[dict[str, Any]] | None = None) -> AsyncGenerator[dict[str, Any], None]  [async]
 ProviderResponse(Generic[T])  [class, @dataclass]

packages/client/src/launchdarkly_ai_server/evaluations/module.py

@@ -11,7 +11,7 @@
 from typing import Any, Literal, cast, overload
 
 from ..lifecycle import _flush_tracer_provider, get_client, init_client
-from ..utils import normalize_mode
+from ..utils import normalize_mode, registration_key
 from .api import (
     DEFAULT_BASE_URI,
     EvaluationsError,
@@ -658,7 +658,7 @@ def _validate_handlers(
             named = [
                 (f"handlers[{index}]", item) for index, item in enumerate(handlers)
             ]
-        seen: dict[tuple[str, str], str] = {}
+        seen: dict[Any, str] = {}
         for name, candidate in named:
             if not callable(candidate):
                 raise EvaluationsError(f"{name} must be callable")
@@ -669,12 +669,14 @@ def _validate_handlers(
                     "create_handler() or a provider package's create_*_handler() "
                     "so that it can be matched to a config's provider and mode."
                 )
-            if provides_for in seen:
+            # Two scoped wildcards of one mode are distinct when their provider lists differ.
+            identity = registration_key(candidate)
+            if identity in seen:
                 raise EvaluationsError(
-                    f"{seen[provides_for]} and {name} both provide for "
+                    f"{seen[identity]} and {name} both provide for "
                     f"{provides_for!r}. Pass one handler for each provider and mode."
                 )
-            seen[provides_for] = name
+            seen[identity] = name
         return [candidate for _, candidate in named]
 
     @staticmethod

packages/client/src/launchdarkly_ai_server/judges.py

@@ -31,15 +31,21 @@
 from .utils import (
     handler_key,
     normalize_mode,
+    select_mode_handler,
     to_ld_context,
     to_usage_dict,
+    wildcard_covers,
 )
 
 
 def _provider_matches(handler: ProviderHandler, provider: str | None) -> bool:
-    """Returns True when the handler covers the given provider or is a wildcard."""
+    """True when the handler names the provider or its wildcard allowlist contains it."""
     key = handler_key(handler)
-    return key is not None and (key[0] == provider or key[0] == "*")
+    if key is None:
+        return False
+    if provider is not None and key[0] == provider:
+        return True
+    return wildcard_covers(handler, provider)
 
 
 def _mode_matches(handler: ProviderHandler, mode: str) -> bool:
@@ -52,35 +58,6 @@ def _mode_matches(handler: ProviderHandler, mode: str) -> bool:
     return key is not None and key[1] == mode
 
 
-def _handler_for(
-    handlers: list[ProviderHandler], provider: str | None, mode: str
-) -> ProviderHandler | None:
-    """Exact ``(provider, mode)`` match, then a same-mode wildcard.
-
-    Modes are compared after normalization, so a ``completion`` tag matches
-    a ``messages`` judge. A wildcard such as LangChain's ``('*', 'messages')``
-    does not hide a handler registered for the judge's own provider.
-    """
-    exact = next(
-        (
-            handler
-            for handler in handlers
-            if (key := handler_key(handler)) is not None and key == (provider, mode)
-        ),
-        None,
-    )
-    if exact is not None:
-        return exact
-    return next(
-        (
-            handler
-            for handler in handlers
-            if (key := handler_key(handler)) is not None and key == ("*", mode)
-        ),
-        None,
-    )
-
-
 logger = logging.getLogger(__name__)
 
 #: The run id of the config a judge graded, on every span of that judge's run.
@@ -179,11 +156,11 @@ async def run_judges(
             # config, collapse messages into a single instructions block.
             judge_handler: ProviderHandler = handler
             collapse_messages = False
-            if handlers:
-                exact = _handler_for(handlers, judge_provider, judge_mode)
+            if handlers is not None:
+                exact = select_mode_handler(handlers, judge_provider, judge_mode)
                 agent_fallback = (
-                    _handler_for(handlers, judge_provider, "agent")
-                    if not exact and judge_mode == "messages"
+                    select_mode_handler(handlers, judge_provider, "agent")
+                    if exact is None and judge_mode == "messages"
                     else None
                 )
                 if exact:
@@ -197,8 +174,10 @@ async def run_judges(
                         handler, "agent"
                     )
                 else:
+                    # No compatible handler — skip rather than calling a provider
+                    # whose allowlist does not include this judge.
                     logger.warning(
-                        "Judge '%s' skipped: no handler provides for provider %r",
+                        "Judge %r skipped: no handler covers provider %r",
                         judge_key,
                         judge_provider,
                     )
@@ -379,11 +358,11 @@ async def _build_judge_tasks(
             )
 
             collapse_messages = False
-            if handlers:
-                exact = _handler_for(handlers, judge_provider, judge_mode)
+            if handlers is not None:
+                exact = select_mode_handler(handlers, judge_provider, judge_mode)
                 agent_fallback = (
-                    _handler_for(handlers, judge_provider, "agent")
-                    if not exact and judge_mode == "messages"
+                    select_mode_handler(handlers, judge_provider, "agent")
+                    if exact is None and judge_mode == "messages"
                     else None
                 )
                 if exact:
@@ -396,6 +375,11 @@ async def _build_judge_tasks(
                     )
                 else:
                     # No compatible handler — skip, same as run_judges.
+                    logger.warning(
+                        "Judge %r skipped: no handler covers provider %r",
+                        judge_key,
+                        judge_provider,
+                    )
                     continue
 
             evaluation_metric_key = (
@@ -448,15 +432,20 @@ async def run_judge(
     report_usage("client.runJudge")
     from .tracking import execute_and_track
 
-    exact = _handler_for(handlers, task.judge_provider, task.judge_mode)
+    exact = select_mode_handler(handlers, task.judge_provider, task.judge_mode)
     agent_fallback = (
-        _handler_for(handlers, task.judge_provider, "agent")
-        if task.judge_mode == "messages" and not exact
+        select_mode_handler(handlers, task.judge_provider, "agent")
+        if task.judge_mode == "messages" and exact is None
         else None
     )
 
     judge_handler = exact or agent_fallback
     if judge_handler is None:
+        logger.warning(
+            "Judge %r skipped: no handler covers provider %r",
+            task.config_key,
+            task.judge_provider,
+        )
         return None
 
     # Defensive: a task serialized before outputFormat was stripped at build time

packages/client/src/launchdarkly_ai_server/registry.py

@@ -5,7 +5,7 @@
 from typing import Any
 
 from .types import NativeTool, ProviderHandler
-from .utils import handler_key
+from .utils import registration_key
 
 logger = logging.getLogger(__name__)
 
@@ -42,15 +42,15 @@ def register(
     ) -> None:
         if handlers:
             for handler in handlers:
-                key = handler_key(handler)
+                key = registration_key(handler)
                 if key is not None:
-                    # Same identity as select_handler / evaluations: provider plus
-                    # normalized mode (completion and messages collide).
+                    # Identity is provides_for plus the provider set. Two scoped
+                    # wildcards of the same mode stay registered when their lists differ.
                     existing_idx = next(
                         (
                             i
                             for i, h in enumerate(self._handlers)
-                            if handler_key(h) == key
+                            if registration_key(h) == key
                         ),
                         None,
                     )

packages/client/src/launchdarkly_ai_server/types.py

@@ -156,6 +156,31 @@ class HandlerStreamEvent:
 ]
 
 
+def normalize_providers(providers: Sequence[str] | None) -> tuple[str, ...] | None:
+    """
+    Validates an optional wildcard allowlist.
+
+    ``None`` means the handler is unscoped. A list must contain at least one
+    name and is stored as a tuple in first-seen order, with repeats dropped.
+    Names are compared to ``config.provider.name`` exactly, so a name with
+    surrounding whitespace is rejected rather than trimmed.
+    """
+    if providers is None:
+        return None
+    if (
+        isinstance(providers, (str, bytes))
+        or len(providers) == 0
+        or any(
+            not isinstance(name, str) or name == "" or name != name.strip()
+            for name in providers
+        )
+    ):
+        raise ValueError(
+            "providers must be a non-empty list of provider names without surrounding whitespace"
+        )
+    return tuple(dict.fromkeys(providers))
+
+
 class ProviderHandler:
     """
     Callable handler with provider metadata.
@@ -164,6 +189,7 @@ class ProviderHandler:
     - ``__call__`` — blocking invocation
     - ``stream``   — optional async-generator streaming (may be ``None``)
     - ``provides_for`` — ``(provider_name, mode)`` tuple or ``None``
+    - ``providers`` — optional allowlist consulted only when ``provides_for`` names ``"*"``
     - ``capture_content`` — the effective content-capture decision for this handler
 
     ``capture_content`` is declared here so the client core can apply the handler's own content
@@ -174,18 +200,22 @@ class ProviderHandler:
     """
 
     provides_for: tuple[str, Literal["agent", "messages"]] | None
+    providers: tuple[str, ...] | None
 
     def __init__(
         self,
         fn: _HandlerFn,
         provides_for: tuple[str, Literal["agent", "messages"]] | None = None,
         stream_fn: _StreamFn | None = None,
         capture_content: bool | None = None,
+        *,
+        providers: Sequence[str] | None = None,
     ) -> None:
         self._fn = fn
         self.provides_for = provides_for
         self._stream_fn = stream_fn
         self._capture_content_override = capture_content
+        self.providers = normalize_providers(providers)
 
     @property
     def capture_content(self) -> bool: