feat!: Add AI online evaluations (Judge, Evaluator, IRunner)

[mattrmc1]

BEGIN_COMMIT_OVERRIDE
feat: Add AI online evaluations (Judge, Evaluator, IRunner)
feat!: JudgeResult defaults for sampled and success changed from true to false per AIEVALS spec
END_COMMIT_OVERRIDE

Summary

Adds AI online evaluations to LaunchDarkly.ServerSdk.Ai. A caller that supplies a runnerFactory to LdAiClient gets automatic judge evaluation wired into every CompletionConfig and AgentConfig — each returned config carries an Evaluator that can score model output against the judges declared in the flag's judgeConfiguration. When no runnerFactory is provided (or no judges are configured), configs receive a noop Evaluator so callers never need null checks.

Implements the AIEVALS and AIRUNNER specs (sections 1.1–1.4). createJudge (AIEVALS 1.2) is intentionally omitted per .NET/Java convention — judges are created internally by the SDK, not by user code.

New public types

// Provider-facing runner interface (AIRUNNER 1.2)
public interface IRunner
{
    Task<RunnerResult> RunAsync(string input,
        IReadOnlyDictionary<string, object> outputType = null);
}

// Runner return type (AIRUNNER 1.3)
public sealed record RunnerResult(
    string Content,
    AiMetrics Metrics,
    object Raw = null,
    IReadOnlyDictionary<string, object> Parsed = null);

// Evaluation orchestrator (AIEVALS 1.4)
public sealed class Evaluator
{
    public static Evaluator Noop();
    public Task<IReadOnlyList<JudgeResult>> EvaluateAsync(string input, string output);
}

// Single-judge executor (AIEVALS 1.1)
public sealed class Judge
{
    public LdAiJudgeConfig Config { get; }
    public IRunner Runner { get; }
    public Task<JudgeResult> EvaluateAsync(string input, string output, double? samplingRate = null);
    public Task<JudgeResult> EvaluateMessagesAsync(
        IReadOnlyList<LdAiConfigTypes.Message> messages,
        RunnerResult runnerResult, double? samplingRate = null);
}

LdAiClient changes

// New optional parameter
public LdAiClient(ILaunchDarklyClient client,
    Func<LdAiJudgeConfig, IRunner> runnerFactory = null);

When runnerFactory is non-null, ConfigFactory.BuildEvaluator iterates the flag's judgeConfiguration, evaluates each judge key as a flag variation, creates a Judge + IRunner pair per enabled judge, and attaches the resulting Evaluator to the config. Disabled judges, null runners, and initialization exceptions are logged and skipped — no single judge failure prevents the others from being built.

LdAiConfig base class

All config types (LdAiCompletionConfig, LdAiAgentConfig, LdAiJudgeConfig) now carry an Evaluator property via the base class. LdAiJudgeConfig always receives Evaluator.Noop() (judges don't evaluate themselves).

JudgeResult changes

Updated to match AIEVALS 1.3.1 defaults:

Field	Before	After
MetricKey	required	optional (default null)
Score	required	optional (default 0.0)
Sampled	default true	default false
Success	default true	default false
ErrorMessage	—	new field
Reasoning	—	new field

Null safety (ref: java-core#175 discussion)

BuildEvaluator handles every failure mode without throwing:

• runnerFactory == null or empty judgeConfiguration → Evaluator.Noop()
• Runner factory returns null → warn + skip judge
• Judge config disabled → warn + skip judge
• Any exception during judge init → warn + skip judge
• Missing evaluationMetricKey → JudgeResult(success: false, errorMessage: ...)
• Score out of [0, 1] range → JudgeResult(success: false, errorMessage: ...)
• Sampling rate NaN/Infinity/negative/> 1.0 → normalized to safe bounds

Judge constructor still uses ArgumentNullException guards, but these are never hit in practice because BuildEvaluator validates inputs before construction.

Migration

None required. The LdAiClient constructor gains an optional runnerFactory parameter (default null) — existing callers are unaffected. JudgeResult default changes are source-compatible (all parameters are now optional with safe defaults). No members removed or renamed.

Test plan

• dotnet test pkgs/sdk/server-ai/test/LaunchDarkly.ServerSdk.Ai.Tests.csproj --framework net8.0 passes
• JudgeTest (435 lines) covers: successful evaluation with score/reasoning extraction, sampling skip path, samplingRate edge cases (NaN, negative, > 1.0), runner exception handling, missing evaluationMetricKey validation, out-of-range score rejection with errorMessage, evaluateMessages formatting, null/empty message handling
• EvaluatorTest (210 lines) covers: noop returns empty list, multi-judge execution with per-judge sampling, missing judge key logs warning and skips, noop does not log warnings
• LdAiCompletionConfigTest (189 lines added) covers: Evaluator attached when runnerFactory provided, noop Evaluator when no runner factory, noop when judgeConfiguration is empty, disabled judge skipped, null runner skipped
• LdAiJudgeConfigTest covers: JudgeResult ErrorMessage and Reasoning fields, judge config always receives noop Evaluator
• LdAiConfigTrackerTest covers: TrackJudgeResult with new optional fields

---

Note

Medium Risk
New optional API is backward compatible, but judge evaluation triggers extra model calls and changes JudgeResult default semantics for code that relied on implicit success/sampled=true.

Overview
Adds online AI evaluations to the server AI SDK: completion and agent configs can run configured judges against model input/output when callers supply a runnerFactory on LdAiClient.

LdAiClient gains an optional Func<LdAiJudgeConfig, IRunner> runnerFactory. ConfigFactory uses it in new BuildEvaluator logic: for each entry in judgeConfiguration, it fetches the judge flag, builds a Judge + IRunner, and attaches an Evaluator to completion/agent configs (including default/fallback paths). Missing factory, empty judges, disabled judges, null runners, and init errors degrade to Evaluator.Noop() or skipped judges with logging—no throw. Judge prompt variables from the parent config are passed through; agent graph builds pass graphKey for judge tracking.

New public surface: IRunner, RunnerResult, Judge, Evaluator. LdAiConfig.Evaluator is always non-null (noop when unused). JudgeResult gains optional ErrorMessage / Reasoning and spec-aligned defaults (Sampled/Success default false). samplingRate on judge config is double? (omit = 100%); parsing and default ToLdValue only emit rate when set.

Judge runs the model via structured score/reasoning schema, sampling, score validation, and metrics via the config tracker; it does not call TrackJudgeResult (caller responsibility). Extensive JudgeTest / EvaluatorTest and config integration tests added.

^{Reviewed by Cursor Bugbot for commit 2562911. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Judge sampling ignores interpolate flag

Medium Severity

BuildEvaluator always calls BuildJudgeConfig with the default interpolate: true, even when the parent config was built with interpolate: false (template APIs). Nested judge configs then run Mustache interpolation (including ldctx) while completion/agent instructions and messages stay as raw templates, so template configs and runnerFactory output can disagree with the documented template behavior.

Additional Locations (2)

• pkgs/sdk/server-ai/src/Config/ConfigFactory.cs#L267-L308
• pkgs/sdk/server-ai/src/LdAiClient.cs#L156-L165

 

^{Reviewed by Cursor Bugbot for commit 6f7d88d. Configure here.}

[mattrmc1]

Not a bug. Attached judges are operational -- they execute against a model provider via IRunner.RunAsync and require interpolated prompts to function. A judge receiving raw {{variable}} tokens would produce nonsense. When fetching a judge directly with interpolation: false, the raw {{variable}} tokens will return correctly

[mattrmc1]

Messages when fetching toxicity judge as template:

[
    {
        "content": "You are a business information accuracy and safety expert. Evaluate....",
        "role": "system"
    },
    {
        "content": "MESSAGE HISTORY:\n{{message_history}}\n",
        "role": "assistant"
    },
    {
        "content": "RESPONSE TO EVALUATE:\n{{response_to_evaluate}}\n",
        "role": "user"
    }
]

[cursor]

Duplicate judge keys run twice

Low Severity

If judgeConfiguration.judges lists the same key more than once, BuildEvaluator stores one Judge in a dictionary but builds filteredConfig from the full list without deduplicating. EvaluateAsync then invokes the same judge once per duplicate entry and returns multiple results for one logical judge.

Additional Locations (1)

• pkgs/sdk/server-ai/src/Config/ConfigFactory.cs#L305-L307

 

^{Reviewed by Cursor Bugbot for commit 6f7d88d. Configure here.}

[mattrmc1]

Is this actually a bug? Should the SDK be defensive about this or faithfully evaluate the list it receives (even if duplicates are present)?

cc @jsonbailey

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 4 total unresolved issues (including 3 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2562911. Configure here.}

[cursor]

Shared Random not thread-safe

Medium Severity

Evaluator stores one Random instance and passes it into every concurrent Judge.EvaluateAsync call for sampling. System.Random is not thread-safe, so parallel EvaluateAsync on the same config can corrupt sampling state and yield incorrect skip/run decisions or unstable behavior under load.

 

^{Reviewed by Cursor Bugbot for commit 2562911. Configure here.}