feat: routing, fallback & rate-limit survival (Phase 4)

.env.example

@@ -22,6 +22,17 @@ TRACE_DB_PATH=./traces.db
 # Optional: also export OpenTelemetry spans to an OTLP/HTTP collector (e.g. Jaeger).
 # OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318/v1/traces
 
+# ── Routing, fallback & rate-limit survival ──────────────────────────
+# Per-attempt timeout before a candidate is aborted and retried/failed over.
+REQUEST_TIMEOUT_MS=30000
+# Retries per candidate on a retryable error (429/5xx/timeout) before failing over.
+MAX_RETRIES=2
+# Default per-provider requests-per-minute throttle (0 = unlimited). Per-provider
+# overrides and the "auto" tier list / fallback chain live in sentinel.config.json.
+DEFAULT_RPM=0
+# Max time the throttle may pace a single candidate before skipping to the next.
+THROTTLE_MAX_WAIT_MS=2000
+
 # ── Semantic cache (embeds prompts locally; serves similar repeats) ───
 CACHE_ENABLED=true
 # Cosine-similarity threshold for a cache hit (0–1; higher = stricter).

README.md

@@ -2,7 +2,7 @@
 
 A self-hostable **verifying LLM gateway** — a drop-in, OpenAI-compatible proxy that **routes** (cheapest capable model + fallback), **semantically caches**, and **verifies** (deterministic guardrails inline + a local Ollama judge) every LLM call, with full OpenTelemetry tracing. Unlike after-the-fact observability tools, it can flag or block a bad response _before it returns_.
 
-> 🚧 **Early development.** Product spec in [`PRP_SPEC.md`](./PRP_SPEC.md), phased build in [`ROADMAP.md`](./ROADMAP.md), contributor/agent guidance in [`CLAUDE.md`](./CLAUDE.md). Currently at **Phase 3 — semantic cache** (routing/fallback and verification land in later phases).
+> 🚧 **Early development.** Product spec in [`PRP_SPEC.md`](./PRP_SPEC.md), phased build in [`ROADMAP.md`](./ROADMAP.md), contributor/agent guidance in [`CLAUDE.md`](./CLAUDE.md). Currently at **Phase 4 — routing, fallback & rate-limit survival** (inline verification lands in later phases).
 
 ## What works today (Phase 1)
 
@@ -73,11 +73,15 @@ Each provider sets a `baseUrl` (or `baseUrlEnv` to read it from the environment)
     }
   },
   "models": { "llama3.2": "ollama", "gpt-4o-mini": "openai", "llama-3.3-70b-versatile": "groq" },
-  "defaultProvider": "ollama"
+  "defaultProvider": "ollama",
+  "routing": {
+    "tiers": ["llama3.2", "gpt-4o-mini", "llama-3.3-70b-versatile"],
+    "fallback": ["llama3.2"]
+  }
 }
 ```
 
-Because almost every provider speaks the OpenAI API, **bringing your own key is just config** — add a provider entry and a model route, drop the key in `.env`, done.
+A provider may also set an `rpm` (requests-per-minute) ceiling; the optional `routing` block configures cost-aware routing (see below). Because almost every provider speaks the OpenAI API, **bringing your own key is just config** — add a provider entry and a model route, drop the key in `.env`, done.
 
 ### Bring your own Ollama
 
@@ -101,6 +105,16 @@ Traces are **metadata only** — no prompt or response bodies are stored, and AP
 
 Sentinel **semantically caches** responses: it embeds each prompt locally (Ollama `nomic-embed-text`) and, when a new request is similar enough to a recent one (cosine ≥ `CACHE_SIMILARITY_THRESHOLD`, default `0.92`), serves the stored answer **without calling the provider** — replaying buffered SSE chunks for streamed requests. The cache is **per-tenant** (scoped to the calling API key), bounded (`CACHE_MAX_ENTRIES`, `CACHE_TTL_SECONDS`), and **fails open** — any embedding error simply falls through to the provider. Cache hits are visible in traces (`GET /traces?cacheHit=true`). Disable with `CACHE_ENABLED=false`.
 
+## Routing & resilience
+
+Sentinel survives flaky providers and free-tier rate limits instead of passing their failures back to your app. Each request becomes an ordered **candidate chain** that a resilience executor runs in order:
+
+- **Retry + backoff + fallback.** A retryable failure (429, upstream 5xx, timeout, network fault) is retried with exponential backoff up to `MAX_RETRIES`, then **failed over** to the next candidate — additional models you list under `routing.fallback`, ending at a local Ollama model so a request can always be served. Terminal errors (400/401/403/404) fail fast without pointless fallback.
+- **Per-provider throttling.** A token-bucket limiter paces each provider to its configured `rpm` (or `DEFAULT_RPM`), waiting up to `THROTTLE_MAX_WAIT_MS` for headroom before skipping to a less-loaded candidate. This keeps you under a provider's limit rather than discovering it the hard way.
+- **Cost-aware routing (opt-in).** Send `"model": "auto"` and a rules-based classifier picks the **cheapest capable tier** from `routing.tiers` (cheapest first) by prompt complexity, escalating to more capable tiers on failure. Explicit model names behave exactly as before — plus the fallback chain. Drop-in semantics are preserved; nothing is configured by default.
+
+Every routed request records which provider/model actually served it, whether a fallback was used, and the retry count — queryable via `GET /traces?fallbackUsed=true` (and `?routedProvider=`). Streaming requests fail over up to the first chunk; once the SSE response is committed, a mid-stream error is surfaced as an inline event.
+
 ## Development
 
 ```bash

SECURITY_REVIEW_LOG.md

@@ -49,7 +49,7 @@ Each item: assume an attacker is actively trying it. Tick only when a test prove
 ### 7. Availability / DoS
 
 - [ ] Sentinel's own rate-limits/quotas protect it from a runaway client.
-- [ ] Bounded queues/timeouts; a slow provider cannot exhaust the event loop or memory.
+- [x] Bounded queues/timeouts; a slow provider cannot exhaust the event loop or memory.
 
 ### 8. Supply chain
 
@@ -64,11 +64,12 @@ Each item: assume an attacker is actively trying it. Tick only when a test prove
 
 ## Review log
 
-| ID     | Date       | Area (1–8) | Change / PR                    | Risk                                                                         | Severity | Status    | Mitigation / notes                                                                                                                                                                                                                                                                                                                                                                                |
-| ------ | ---------- | ---------- | ------------------------------ | ---------------------------------------------------------------------------- | -------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| —      | 2026-06-27 | —          | Initial context docs (no code) | n/a                                                                          | n/a      | n/a       | Baseline.                                                                                                                                                                                                                                                                                                                                                                                         |
-| SR-001 | 2026-06-27 | 1, 3, 4    | Phase 1 pass-through proxy     | Unauthenticated access; key/secret leakage; SSRF via request-controlled URLs | high     | mitigated | Bearer Sentinel-key auth required — 401 on missing/invalid (tested). Provider keys read from env via `apiKeyEnv`, never hard-coded, never returned to clients. Provider base URLs come only from the config file, never request input. `authorization`/`x-api-key` redacted in request logs via pino `redact` (configured; an explicit log-assertion test is still TODO — box 1.2 left unticked). |
-| SR-002 | 2026-06-27 | 3, 6       | Phase 2 tracing & persistence  | Unauthorized trace access; secrets/PII in stored traces                      | high     | mitigated | `GET /traces` gated by a separate `SENTINEL_ADMIN_KEY` — 401 without it (tested), distinct from client keys. Traces are metadata-only (model, provider, tokens, latency, status) — no prompt/response bodies persisted. API keys stored as a SHA-256 hash, never raw. Per-key trace scoping deferred (admin-only for now).                                                                        |
-| SR-003 | 2026-06-27 | 5          | Phase 3 semantic cache         | Cross-tenant cache leakage; wrong-answer collisions                          | high     | mitigated | Cache entries are bucketed per-tenant by API-key hash — no cross-tenant hits (tested). The bucket also keys on model/temperature/max_tokens/stream/embed-model, so requests that change the answer never collide; semantic matching happens only within a bucket above a conservative threshold (0.92, configurable). Cache fails open (embed errors → miss).                                     |
+| ID     | Date       | Area (1–8) | Change / PR                    | Risk                                                                                                                                                         | Severity | Status    | Mitigation / notes                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| ------ | ---------- | ---------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| —      | 2026-06-27 | —          | Initial context docs (no code) | n/a                                                                                                                                                          | n/a      | n/a       | Baseline.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+| SR-001 | 2026-06-27 | 1, 3, 4    | Phase 1 pass-through proxy     | Unauthenticated access; key/secret leakage; SSRF via request-controlled URLs                                                                                 | high     | mitigated | Bearer Sentinel-key auth required — 401 on missing/invalid (tested). Provider keys read from env via `apiKeyEnv`, never hard-coded, never returned to clients. Provider base URLs come only from the config file, never request input. `authorization`/`x-api-key` redacted in request logs via pino `redact` (configured; an explicit log-assertion test is still TODO — box 1.2 left unticked).                                                                                                                                                                       |
+| SR-002 | 2026-06-27 | 3, 6       | Phase 2 tracing & persistence  | Unauthorized trace access; secrets/PII in stored traces                                                                                                      | high     | mitigated | `GET /traces` gated by a separate `SENTINEL_ADMIN_KEY` — 401 without it (tested), distinct from client keys. Traces are metadata-only (model, provider, tokens, latency, status) — no prompt/response bodies persisted. API keys stored as a SHA-256 hash, never raw. Per-key trace scoping deferred (admin-only for now).                                                                                                                                                                                                                                              |
+| SR-003 | 2026-06-27 | 5          | Phase 3 semantic cache         | Cross-tenant cache leakage; wrong-answer collisions                                                                                                          | high     | mitigated | Cache entries are bucketed per-tenant by API-key hash — no cross-tenant hits (tested). The bucket also keys on model/temperature/max_tokens/stream/embed-model, so requests that change the answer never collide; semantic matching happens only within a bucket above a conservative threshold (0.92, configurable). Cache fails open (embed errors → miss).                                                                                                                                                                                                           |
+| SR-004 | 2026-06-27 | 7          | Phase 4 routing & fallback     | Self-inflicted DoS: unbounded retries/fallback amplify load; a slow or hostile provider stalls the event loop; throttle/router as a new untrusted-input path | med      | mitigated | Each attempt is bounded by a per-attempt `AbortController` timeout (`REQUEST_TIMEOUT_MS`) so a slow provider can't hang the loop (tested). Retries are capped (`MAX_RETRIES`) and fallback walks a finite, **config-defined** candidate chain — request input never adds providers/URLs (no new SSRF surface). A per-provider token bucket paces outbound calls to each provider's `rpm`, protecting upstreams and avoiding self-induced 429 storms. Terminal 4xx errors fail fast without amplification. Inbound per-client rate-limiting (box 7.1) is still deferred. |
 
 > Add a row per high-risk change. Status ∈ {open, mitigated, accepted}. Severity ∈ {low, med, high, critical}.

packages/gateway/src/config.ts

@@ -17,6 +17,10 @@ export interface ServerEnv {
   cacheMaxEntries: number;
   ollamaBaseUrl: string;
   embedModel: string;
+  requestTimeoutMs: number;
+  maxRetries: number;
+  defaultRpm: number;
+  throttleMaxWaitMs: number;
 }
 
 const serverEnvSchema = z.object({
@@ -35,6 +39,10 @@ const serverEnvSchema = z.object({
   CACHE_MAX_ENTRIES: z.coerce.number().int().positive().default(1000),
   OLLAMA_BASE_URL: z.string().default('http://localhost:11434/v1'),
   EMBED_MODEL: z.string().default('nomic-embed-text'),
+  REQUEST_TIMEOUT_MS: z.coerce.number().int().nonnegative().default(30_000),
+  MAX_RETRIES: z.coerce.number().int().nonnegative().default(2),
+  DEFAULT_RPM: z.coerce.number().int().nonnegative().default(0),
+  THROTTLE_MAX_WAIT_MS: z.coerce.number().int().nonnegative().default(2_000),
 });
 
 /** Reads and validates the process environment Sentinel needs to run. */
@@ -62,6 +70,10 @@ export function loadServerEnv(env: NodeJS.ProcessEnv): ServerEnv {
     cacheMaxEntries: parsed.data.CACHE_MAX_ENTRIES,
     ollamaBaseUrl: parsed.data.OLLAMA_BASE_URL,
     embedModel: parsed.data.EMBED_MODEL,
+    requestTimeoutMs: parsed.data.REQUEST_TIMEOUT_MS,
+    maxRetries: parsed.data.MAX_RETRIES,
+    defaultRpm: parsed.data.DEFAULT_RPM,
+    throttleMaxWaitMs: parsed.data.THROTTLE_MAX_WAIT_MS,
   };
 }
 
@@ -73,16 +85,23 @@ const providerConfigSchema = z
     baseUrl: z.string().url().optional(),
     baseUrlEnv: z.string().min(1).optional(),
     apiKeyEnv: z.string().min(1).optional(),
+    rpm: z.coerce.number().int().nonnegative().optional(),
   })
   .refine((p) => (p.baseUrl === undefined) !== (p.baseUrlEnv === undefined), {
     message: 'set exactly one of "baseUrl" or "baseUrlEnv"',
   });
 
+const routingConfigSchema = z.object({
+  tiers: z.array(z.string().min(1)).optional(),
+  fallback: z.array(z.string().min(1)).optional(),
+});
+
 const sentinelConfigSchema = z
   .object({
     providers: z.record(z.string(), providerConfigSchema),
     models: z.record(z.string(), z.string()),
     defaultProvider: z.string().optional(),
+    routing: routingConfigSchema.optional(),
   })
   .superRefine((cfg, ctx) => {
     const names = new Set(Object.keys(cfg.providers));
@@ -106,13 +125,22 @@ export interface ResolvedProvider {
   name: string;
   baseUrl: string;
   apiKey: string | undefined;
+  /** Per-provider requests-per-minute limit for the throttle (omitted = unlimited). */
+  rpm?: number;
+}
+
+/** Routing rules (tier list + fallback chain) from the config file. */
+export interface ResolvedRouting {
+  tiers?: string[] | undefined;
+  fallback?: string[] | undefined;
 }
 
 export interface ResolvedConfig {
   providers: Map<string, ResolvedProvider>;
   /** model name → provider name */
   models: Map<string, string>;
   defaultProvider: string | undefined;
+  routing?: ResolvedRouting;
 }
 
 export interface LoadConfigOptions {
@@ -149,13 +177,14 @@ export function loadConfig(options: LoadConfigOptions): ResolvedConfig {
   for (const [name, p] of Object.entries(parsed.data.providers)) {
     const baseUrl = p.baseUrl ?? readEnvOrThrow(options.env, p.baseUrlEnv, name);
     const apiKey = p.apiKeyEnv === undefined ? undefined : options.env[p.apiKeyEnv];
-    providers.set(name, { name, baseUrl, apiKey });
+    providers.set(name, { name, baseUrl, apiKey, ...(p.rpm !== undefined ? { rpm: p.rpm } : {}) });
   }
 
   return {
     providers,
     models: new Map(Object.entries(parsed.data.models)),
     defaultProvider: parsed.data.defaultProvider,
+    ...(parsed.data.routing ? { routing: parsed.data.routing } : {}),
   };
 }
 

packages/gateway/src/index.ts

@@ -3,11 +3,30 @@
 export const SENTINEL_VERSION = '0.1.0';
 
 export { buildServer } from './server.js';
-export type { ServerDeps } from './server.js';
+export type { ServerDeps, RoutingDeps } from './server.js';
 export { loadConfig, loadServerEnv } from './config.js';
-export type { ResolvedConfig, ResolvedProvider, ServerEnv, LoadConfigOptions } from './config.js';
+export type {
+  ResolvedConfig,
+  ResolvedProvider,
+  ResolvedRouting,
+  ServerEnv,
+  LoadConfigOptions,
+} from './config.js';
 export { createRegistry } from './providers/registry.js';
 export type { ProviderRegistry } from './providers/registry.js';
+export { createRouter } from './routing/router.js';
+export type { Router, Candidate, RoutingConfig } from './routing/router.js';
+export { runChat, openStream } from './routing/executor.js';
+export type { RouteOutcome, ExecutorOptions, StreamHandle } from './routing/executor.js';
+export { isRetryable } from './routing/retryable.js';
+export { classifyTier } from './routing/classifier.js';
+export { createTokenBucket, createBucketRegistry } from './throttle/token-bucket.js';
+export type {
+  TokenBucket,
+  TokenBucketOptions,
+  BucketRegistry,
+  BucketRegistryOptions,
+} from './throttle/token-bucket.js';
 export { createOpenAICompatibleProvider } from './providers/openai-compatible.js';
 export type { Provider, FetchLike } from './providers/types.js';
 export {