Repository navigation
feat(server): Jev-compatible /v1/systemone single-forward decision endpoint (#708) - #709
Conversation
…on endpoint (#708) State + typed questions (noul/choice/score) -> typed answers with probabilities, one prefill per question and no decode loop: the Tev1 {state,question,options} contract, option-letter logits at the first generated position, restricted softmax. Wire shape matches TypeSafe's API, OpenJev servers and the Microsoft Agent Framework TypeSafe provider (parser rules replayed in a test). Verified live on Tev1-4B Vulkan (direct and scheduler paths): 12 days -> P(in window)=0.994, 45 days -> 0.001, ~65 ms/question. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The new endpoint runs a model forward pass through the generator/scheduler on every backend and has one convention-deviation finding (prompt-length validated only after inference), so it warrants human review before approval.
Review effort: Balanced
Findings: 1
Open (1)
What changed in this PR
This PR adds a Jev-compatible POST /v1/systemone decision endpoint (#708). Jev (TypeSafe AI) is a non-autoregressive decision model (state + typed questions → typed answers with probabilities in one pass); dotLLM imitates that wire contract on an ordinary instruction-tuned LM head by running one prefill per question (no decode loop), capturing the option-letter logits at the first generated position via an ILogitProcessor, and renormalising them into per-option probabilities. The capture-processor design lets the same path run unchanged through both the direct generator and the continuous-batch scheduler on every backend.
Changes:
- New engine component
DecisionEvaluator+DecisionMath(Tev1 prompt contract, closed thinking block, single-forward logit read-out, softmax/confidence/expected-index math). - New server surface:
SystemOneModelsDTOs,SystemOneEndpoint(validation, error envelope, scheduler/direct routing, rate-limit reconciliation), context + endpoint + metered-path registration. - 16 unit tests (incl. an Agent Framework parser replay) and
docs/SERVER.mddocumentation.
| File | Description |
|---|---|
src/DotLLM.Engine/Decisions/DecisionEvaluator.cs |
Core single-forward decision read-out, prompt builder, and DecisionMath. |
src/DotLLM.Server/Endpoints/SystemOneEndpoint.cs |
HTTP endpoint: request validation, per-question scoring, scheduler/direct routing, error handling. |
src/DotLLM.Server/Models/SystemOneModels.cs |
Request/answer/usage/response DTOs with explicit JSON property names. |
src/DotLLM.Server/ServerJsonContext.cs |
Registers the new DTOs in the source-gen JSON context. |
src/DotLLM.Server/EndpointExtensions.cs |
Maps the new endpoint during app startup. |
tests/DotLLM.Tests.Unit/Server/SystemOneEndpointTests.cs |
Scripted-runner unit tests covering math, prompts, answers, validation, and parser compliance. |
tests/DotLLM.Tests.Unit/Server/MeteredPathPolicyTests.cs |
Adds /v1/systemone to the metered-route guards. |
docs/SERVER.md |
Documents the endpoint, question types, and limitations. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| var dist = await evaluator.ScoreAsync(state, instructions, options, ct).ConfigureAwait(false); | ||
| if (dist.PromptTokens > maxSequenceLength) | ||
| throw new SystemOneException(422, $"The prompt for question '{id}' is {dist.PromptTokens} tokens, over the model's context of {maxSequenceLength}.", "state"); |
Closes #708
Jev (TypeSafe AI) is non-autoregressive: state + typed questions -> typed answers with probabilities in one pass. Tev1 only imitates the contract on a normal LM head. This adds
POST /v1/systemonewith Jev's wire shape (TypeSafe API / OpenJev / Microsoft Agent FrameworkMicrosoft.Agents.AI.TypeSafe):noul,choice,scorequestions; one prefill per question, no decode loop, option-letter logits at the first generated position renormalised into probabilities (Jev's1 - H/ln Kconfidence).DecisionEvaluator(Tev1 prompt contract, thinking closed, logit capture via anILogitProcessor, so it runs unchanged through the direct generator and the continuous-batch scheduler on every backend).Limitations (documented): probabilities are the model's own restricted softmax, not calibrated like Jev's; score is an expected level index.
🤖 Generated with Claude Code