feat(cache): pgvector semantic cache backend (Stage 4b)

[moonming]

Background

The exact-match cache requires byte-identical prompts to hit. Real workloads see large variation in inputs that are semantically the same — minor whitespace / punctuation differences, paraphrased instructions, agent-injected timestamps, user IDs in prompt context. On real traffic the exact-match hit rate can hover at 5–15 %.

The CachePolicy resource already carries the wire shape for semantic mode (backend: redis_semantic | qdrant, similarity_threshold, embedding_model) — these fields land in cp-api today and the DP currently falls back to the memory backend.

The pgvector DP backend is the cheapest path to first-class semantic cache because we already operate Postgres for kine.

Open work

• DP-side embedder: lazy-initialised client per embedding_model (OpenAI text-embedding-3-small first; provider-agnostic interface for follow-up).
• New aisix-cache::PgVectorCache backend implementing the Cache trait. Schema: (policy_id, embedding vector, response jsonb, created_at) with an HNSW index keyed on embedding.
• chat::dispatch cache gate: when the matched policy's backend == redis_semantic | qdrant (and pgvector is the configured implementation), embed → ANN-search → compare similarity to threshold → hit / miss.
• Telemetry: emit the similarity score on hit, the nearest distance on miss; integrates with the cost-saved telemetry from feat(cache): cost-saved telemetry on cache hit #88 once that lands.
• Migration in cp-api's pgvector schema.
• e2e against a real embedding service (no mocks per CLAUDE.md): paraphrased-prompt set should hit at threshold 0.9, fail at 0.99.

Why this delivers value

1. Hit rate goes up by an order of magnitude on realistic traffic (FAQ / customer-support / agent / RAG-lite). Exact 10 % → semantic 50–70 % is consistent with published numbers for similar systems.
2. Cost savings scale with hit rate — combined with feat(cache): cost-saved telemetry on cache hit #88 cost-saved telemetry, semantic cache becomes the single biggest user-facing dollar number this product reports.
3. Latency improvement. ANN lookup is ms-scale; LLM calls are 100s of ms to seconds. End-user UX wins materially on hits.
4. Tunable accuracy. Operators dial similarity_threshold per policy — strict for legal / compliance, loose for FAQ — without rewriting the cache layer.
5. Differentiation. Most open-source AI gateways ship exact-match cache only. Semantic + multi-tenant policy + observability is the killer feature combo.

Status

PR #86 is open with the initial DP backend and is the work-in-progress home for this issue. Closing this issue when that PR (or its successors) lands and an e2e against a real embedding service is green.

Acceptance criteria

• aisix-cache::PgVectorCache implements the Cache trait, including put_with_ttl.
• Migration adds cache_entries table with vector column and HNSW index.
• chat::dispatch selects the pgvector backend when the matched policy's backend is set accordingly.
• Embedding client is lazy and per-model; no embedding call on cache miss (i.e. embed once, write once).
• e2e test against a real embedder verifies: identical prompt hits, paraphrase above threshold hits, paraphrase below threshold misses, the miss is then cached and the same paraphrase hits on the second call.
• Telemetry: hit events carry similarity_score; miss events carry nearest_distance for tuning.

Out of scope

• Multi-vector / hybrid retrieval (BM25 + vector) — second-iteration concern.
• Redis-Stack redis_semantic backend — separate implementation, can land later under the same trait.