feat(jev): cache, fail over, pin versions, and serve Kev routes on /v1/systemone - #1098
Conversation
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Advanced Run ID: 📒 Files selected for processing (11)
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review. 📝 WalkthroughWalkthroughThe change adds System One permutation and separate-question routes. It updates model resolution and request validation, and adds cached dispatch with provider failover. API documentation, audit classification, guardrail logging, and cached JSON usage extraction also change. ChangesSystem One request flow
Priority: ➖ Normal Estimated code review effort: 4 (Complex) | ~45 minutes Change: Feature Sequence Diagram(s)sequenceDiagram
participant Client
participant SystemOneHandler
participant ResponseCache
participant InferenceOrchestrator
participant JevProvider
Client->>SystemOneHandler: Submit System One request
SystemOneHandler->>ResponseCache: Dispatch with cache context
alt Exact cache hit
ResponseCache-->>SystemOneHandler: Replay cached response
else Cache miss or caching disabled
ResponseCache->>InferenceOrchestrator: Execute passthrough with failover
InferenceOrchestrator->>JevProvider: Send request to eligible target
JevProvider-->>InferenceOrchestrator: Return response or error
InferenceOrchestrator-->>ResponseCache: Return response and answering selector
ResponseCache-->>SystemOneHandler: Return dispatch result
end
SystemOneHandler-->>Client: Proxy response
Merge Risk: ⚪ Minimal · up to The reviewed System One changes have no identified issue requiring resolution before merge. Security Architecture ReviewSecurity architecture risk: 🟡 Moderate · up to New routes can send decision requests to multiple providers and replay prior answers. The reviewed paths retain access checks and restrict eligible providers, but the expanded behavior warrants design review. Retained concerns Security review detailsSecurity Blast Radius
Trust Boundaries and Controls
Resilience and Maintainability Implications
Hardening Proposals
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Docstring CoverageExplanation Docstring coverage is 65.85% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 41 functions across 21 files. (5 skipped: 5 unsupported.)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. A rabbit taps a route anew, Comment |
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
No flows tested, and faced 1 obstacle. Obstacles faced
To reduce obstacles, configure your TREX environment. |
f8cf9f8 to
ec03d7f
Compare
ec03d7f to
f6a573e
Compare
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
|
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…d warn once on dropped guardrail edits
This branch was successfully deployed
2 participants
Follow-up to #1097. Closes the gaps #1097 left open on
/v1/systemone.User-visible impact
Response caching: an identical request is answered from the exact cache (
X-Cache: HIT (exact)) and recorded in usage as a cache hit with its token counts. The semantic cache never serves System One, since a similar state is not the same decision.Failover: a
failovervirtual model moves a request to its next target on an availability error (429/5xxby default), in the target's own System One form. Targets without the API (chat models) are skipped, not called;4xxclient errors do not fail over. Attempts appear in the audit log; usage is recorded under the target that answered; failover answers are not cached.Kev routes:
POST /v1/systemone/permuteand/v1/systemone/separateare native, with virtual models, guardrails, caching, audit, and usage. They are refused for OpenRouter, which answers only the evaluation route.Pinned versions: TypeSafe lists only its aliases but accepts any versioned ID, so
jev-1.13.0now routes without declaring it or settingCONFIGURED_PROVIDER_MODELS_MODE=merge: when named with its provider (jev/jev-1.13.0), bare with exactly onejevprovider, or through a virtual model. This is checked before catalog resolution, which would otherwise refresh the provider's model list on every such request.Misuse on OpenAI routes: a System One model (a
jevmodel or an OpenRouter decision model, directly or through a virtual model) sent to chat,/responses,/v1/messages, or embeddings now gets a400pointing atPOST /v1/systemone, decided from the catalog. Before, OpenRouter's own error told callers to use its/api/alpha/decisions.Quieter guardrail warning: edits a decision request cannot carry (e.g. a chat system prompt) are reported once per kind at warning level, then at debug level, instead of on every request.
Docs: new System One API page under Advanced, listed in the API endpoints index; the Jev / Kev provider page now links there.
Notes
Summary by CodeRabbit
POST /v1/systemonedecision endpoint for supported Jev and OpenRouter models, with provider-specific routing, guardrails, auditing, usage tracking, caching, and failover.