feat: Phase 3 — semantic cache (local embeddings + per-tenant)

[MANVENDRA-github]

What

ROADMAP Phase 3 — semantic cache (PRP C4). Embed each request locally and, when a new request is similar enough to a recent one, serve the stored response without calling the provider — the first real cost/latency win.

• SemanticCache (in-memory, brute-force cosine) behind an interface — swappable to Redis/vector later. An Ollama /embeddings embedder (keyless, reuses the provider fetch pattern) + a cosine util.
• Per-tenant + correct keying — entries are bucketed by API-key hash plus model / temperature / max_tokens / stream / embed-model, so no cross-tenant hits and no collisions across requests that change the answer; semantic matching happens only within a bucket, above a configurable threshold (default 0.92).
• Streaming too — non-streaming responses are cached as JSON; streaming responses are buffered on a miss and replayed as SSE on a hit.
• Fails open — any embed/cache error simply falls through to the provider; the request never breaks.
• Observability — cacheHit is recorded on the span + trace and is queryable via GET /traces?cacheHit=true.
• Config — CACHE_ENABLED, CACHE_SIMILARITY_THRESHOLD, CACHE_TTL_SECONDS, CACHE_MAX_ENTRIES, plus OLLAMA_BASE_URL / EMBED_MODEL. SECURITY_REVIEW_LOG SR-003 (area 5 ticked).

How to verify

pnpm install
pnpm verify     # typecheck + lint + 89 tests, 91% branch coverage

Live smoke test

In-process, real OTel bootstrap + real cache (deterministic stub embedder):

• same prompt twice -> 2nd is served from cache, provider.chat called once;
• GET /traces?cacheHit=true -> returns the hit (cacheHit:true, tokens, hashed key).

(Live Ollama embeddings depend on the machine's Ollama runtime, which currently crashes; if embeddings error in real use the cache fails open and the gateway works unchanged — so the smoke test uses a stub embedder for determinism.)

Notes

• No new runtime deps — the cache is pure TS (cosine + Map).
• set is awaited on a miss for deterministic population; could be backgrounded later to shave miss latency.
• Streaming and non-streaming live in separate buckets, so a prompt asked both ways won't share — a deliberate correctness-over-hit-rate trade.
• Next: Phase 4 (routing, fallback & rate-limit survival).