diff --git a/CHANGELOG.md b/CHANGELOG.md index 86364ad..9944b03 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ All notable changes to Sentinel are documented here. The format follows ### Added - Per-request **cost tracking**: set a `pricing` map (USD per 1K tokens, per model) in `sentinel.config.json` and every trace records a `costUsd`; the dashboard shows total spend, spend saved by the cache, and cost over time. +- **Native Anthropic provider**: set `"type": "anthropic"` on a provider and Sentinel speaks Anthropic's Messages API (`x-api-key`, request/response/stream translation) while clients keep using the OpenAI shape. ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 11fa30f..9d5f84f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,7 +13,7 @@ The product source of truth is `PRP_SPEC.md`. The build order is `ROADMAP.md`. B - Package manager / monorepo: **pnpm** workspaces. - Gateway server: **Fastify** (streaming, hooks, `.inject()` testing). - Schemas & validation: **Zod** at every trust boundary (config, request/response, guardrails). -- Providers: a unified, hand-rolled **OpenAI-compatible** HTTP adapter (`fetch`) — OpenAI, Groq, Gemini's OpenAI endpoint, Ollama, and other OpenAI-API providers. A native **Anthropic** (Messages API) adapter is on the near-term roadmap. +- Providers: a hand-rolled **OpenAI-compatible** HTTP adapter (`fetch`) — OpenAI, Groq, Gemini's OpenAI endpoint, Ollama, and other OpenAI-API providers — plus a native **Anthropic** (Messages API) adapter. Selected per provider by `type` in `sentinel.config.json`. - Local models (keyless): **Ollama** — judge `qwen2.5:7b`, embeddings `nomic-embed-text`. - Cache & rate-limit state: **in-memory** (single-node v1). A **Redis** (`ioredis`) backend is a planned swap behind the cache/throttle interfaces. - Storage (traces, eval results): **better-sqlite3** / in-memory (single-node v1). A **Postgres** backend is a planned swap behind the `TraceStore` interface. diff --git a/README.md b/README.md index 31b1edd..663bd64 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ A self-hostable **verifying LLM gateway** — a drop-in, OpenAI-compatible proxy ## What it does -A drop-in, OpenAI-compatible `POST /v1/chat/completions` endpoint (streaming and non-streaming) that authenticates callers, validates requests, and forwards them to **any OpenAI-compatible provider** you configure — OpenAI, Groq, Mistral, OpenRouter, DeepSeek, xAI, Google Gemini (via its OpenAI endpoint), or a local **Ollama** model. Point your existing OpenAI SDK at Sentinel by changing one line: the base URL. +A drop-in, OpenAI-compatible `POST /v1/chat/completions` endpoint (streaming and non-streaming) that authenticates callers, validates requests, and forwards them to the provider you configure — **any OpenAI-compatible API** (OpenAI, Groq, Mistral, OpenRouter, DeepSeek, xAI, Google Gemini via its OpenAI endpoint, or a local **Ollama** model) or **Anthropic** via its native Messages API. Point your existing OpenAI SDK at Sentinel by changing one line: the base URL. ## Requirements @@ -85,7 +85,7 @@ Each provider sets a `baseUrl` (or `baseUrlEnv` to read it from the environment) } ``` -A provider may also set an `rpm` (requests-per-minute) ceiling; the optional `routing` block configures cost-aware routing (see below). Because almost every provider speaks the OpenAI API, **bringing your own key is just config** — add a provider entry and a model route, drop the key in `.env`, done. +A provider may also set an `rpm` (requests-per-minute) ceiling; the optional `routing` block configures cost-aware routing (see below). Because almost every provider speaks the OpenAI API, **bringing your own key is just config** — add a provider entry and a model route, drop the key in `.env`, done. **Anthropic** is supported natively too: set `"type": "anthropic"` on the provider and Sentinel translates to/from its Messages API (clients keep sending the OpenAI shape). ### Bring your own Ollama @@ -163,7 +163,6 @@ Sentinel v0.1.0 is a **single-node, self-hosted** gateway. Honest boundaries tod - **SQLite (or in-memory) trace storage.** Ideal for single-node; a Postgres backend for multi-instance is planned. - **The async judge needs a local Ollama** with `JUDGE_MODEL` pulled. Without it, judging degrades to `unscored` (never a false pass). The bundled benchmarks run against **mock upstreams**, so the headline catch-rate is the _deterministic guardrail_ rate; the LLM judge is covered by unit tests. - **Inline guardrails apply to non-streaming responses.** Streamed responses are judged from their buffered output _after_ completion (inline blocking of a live stream is on the roadmap). -- **Providers are OpenAI-compatible.** OpenAI, Groq, Gemini's OpenAI endpoint, Ollama, and other OpenAI-API providers work today; a native **Anthropic** (Messages API) adapter is planned. - **Per-request dollar cost requires a `pricing` map** in `sentinel.config.json` (USD per 1K tokens, per model). With it, every trace records a `costUsd` and the dashboard shows spend + cache savings; without it, cost is unknown (`null`) and only request-volume cache savings are visible. ## Development diff --git a/SECURITY_REVIEW_LOG.md b/SECURITY_REVIEW_LOG.md index 8e5e678..8c5661c 100644 --- a/SECURITY_REVIEW_LOG.md +++ b/SECURITY_REVIEW_LOG.md @@ -75,4 +75,6 @@ Each item: assume an attacker is actively trying it. Tick only when a test prove | SR-006 | 2026-06-27 | 1, 4, 7, 8 | Phase 7 hardening & launch | Key leakage to logs; SSRF via upstream redirect; self-inflicted DoS from a runaway client; supply-chain advisories | high | mitigated | Box 1.2: `logRedaction` redacts `authorization`/`x-api-key`; a test proves the header logs as `[redacted]` and the raw key never appears (Fastify's default serializer also omits headers, so keys can't leak by default). Box 4.2: the upstream `fetch` uses `redirect: 'error'` — a malicious provider can't 3xx-redirect the prompt to another host (tested). Box 7.1: a per-API-key inbound token bucket (`CLIENT_RPM`) returns 429 when one key exceeds its budget, without affecting other clients (tested). Box 8.1: CI runs `pnpm audit --prod --audit-level=high` and production deps are clean; the only advisories are dev-only (vitest/vite test tooling, never shipped), triaged and tracked for a vitest 3 upgrade. Boxes 3.2/6.2/8.2 ticked from existing coverage (cache per-tenant isolation tests; React-escaped, metadata-only dashboard; no untrusted post-install scripts). | +| SR-007 | 2026-06-30 | 1, 4 | PR3 native Anthropic adapter | A second outbound auth path (`x-api-key`) and a new response/stream shape: risk of leaking the Anthropic key to logs, following a malicious redirect, or trusting a malformed upstream body | high | mitigated | The Anthropic key is sent only in the outbound `x-api-key` header — never logged (`logRedaction` already redacts inbound `x-api-key`; Fastify omits request headers by default) and never returned to clients; it comes from `ANTHROPIC_API_KEY` via `apiKeyEnv`, never hard-coded. The adapter reuses `redirect: 'error'`, so a malicious provider can't 3xx the prompt to another host (tested), and the base URL still comes only from the config allow-list, never request input. The Anthropic response is parsed through a Zod schema before translation; an unexpected shape ⇒ `UpstreamError` (502), never a malformed pass-through (tested). | + > Add a row per high-risk change. Status ∈ {open, mitigated, accepted}. Severity ∈ {low, med, high, critical}. diff --git a/packages/gateway/src/config.test.ts b/packages/gateway/src/config.test.ts index ec6f5e3..a2531fd 100644 --- a/packages/gateway/src/config.test.ts +++ b/packages/gateway/src/config.test.ts @@ -115,6 +115,34 @@ describe('loadConfig', () => { expect(() => loadConfig({ path: 'x', env, readFile: () => bad })).toThrow(ConfigError); }); + it('resolves the anthropic provider adapter type', () => { + const cfg = JSON.stringify({ + providers: { + claude: { + type: 'anthropic', + baseUrl: 'https://api.anthropic.com/v1', + apiKeyEnv: 'ANTHROPIC_API_KEY', + }, + }, + models: { 'claude-3-5-sonnet': 'claude' }, + }); + const resolved = loadConfig({ + path: 'x', + env: { ANTHROPIC_API_KEY: 'sk-ant' }, + readFile: () => cfg, + }); + expect(resolved.providers.get('claude')?.type).toBe('anthropic'); + expect(resolved.providers.get('claude')?.apiKey).toBe('sk-ant'); + }); + + it('rejects an unknown provider adapter type', () => { + const bad = JSON.stringify({ + providers: { p: { type: 'cohere', baseUrl: 'https://x' } }, + models: {}, + }); + expect(() => loadConfig({ path: 'x', env, readFile: () => bad })).toThrow(ConfigError); + }); + it('throws on invalid JSON', () => { expect(() => loadConfig({ path: 'x', env, readFile: () => '{ not json' })).toThrow(ConfigError); }); diff --git a/packages/gateway/src/config.ts b/packages/gateway/src/config.ts index ccb0e98..b839e09 100644 --- a/packages/gateway/src/config.ts +++ b/packages/gateway/src/config.ts @@ -109,7 +109,7 @@ export function loadServerEnv(env: NodeJS.ProcessEnv): ServerEnv { const providerConfigSchema = z .object({ - type: z.literal('openai-compatible'), + type: z.enum(['openai-compatible', 'anthropic']), baseUrl: z.string().url().optional(), baseUrlEnv: z.string().min(1).optional(), apiKeyEnv: z.string().min(1).optional(), @@ -165,6 +165,8 @@ const sentinelConfigSchema = z export interface ResolvedProvider { name: string; + /** Which adapter serves this provider: a generic OpenAI-compatible HTTP API or Anthropic's Messages API. */ + type: 'openai-compatible' | 'anthropic'; baseUrl: string; apiKey: string | undefined; /** Per-provider requests-per-minute limit for the throttle (omitted = unlimited). */ @@ -229,7 +231,13 @@ export function loadConfig(options: LoadConfigOptions): ResolvedConfig { for (const [name, p] of Object.entries(parsed.data.providers)) { const baseUrl = p.baseUrl ?? readEnvOrThrow(options.env, p.baseUrlEnv, name); const apiKey = p.apiKeyEnv === undefined ? undefined : options.env[p.apiKeyEnv]; - providers.set(name, { name, baseUrl, apiKey, ...(p.rpm !== undefined ? { rpm: p.rpm } : {}) }); + providers.set(name, { + name, + type: p.type, + baseUrl, + apiKey, + ...(p.rpm !== undefined ? { rpm: p.rpm } : {}), + }); } return { diff --git a/packages/gateway/src/providers/anthropic.test.ts b/packages/gateway/src/providers/anthropic.test.ts new file mode 100644 index 0000000..ce1f6cc --- /dev/null +++ b/packages/gateway/src/providers/anthropic.test.ts @@ -0,0 +1,226 @@ +import { describe, it, expect, vi } from 'vitest'; +import { createAnthropicProvider } from './anthropic.js'; +import { UpstreamError } from '../errors.js'; +import type { ChatCompletionRequest } from '../schemas.js'; + +const request: ChatCompletionRequest = { + model: 'claude-3-5-sonnet', + messages: [ + { role: 'system', content: 'Be brief.' }, + { role: 'user', content: 'hi' }, + ], + max_tokens: 100, + temperature: 0.5, +}; + +function jsonResponse(body: unknown): Response { + return new Response(JSON.stringify(body), { status: 200 }); +} + +function streamResponse(chunks: string[]): Response { + const body = new ReadableStream({ + start(controller) { + const encoder = new TextEncoder(); + for (const chunk of chunks) controller.enqueue(encoder.encode(chunk)); + controller.close(); + }, + }); + return new Response(body, { status: 200 }); +} + +const reply = { + id: 'msg_1', + type: 'message', + role: 'assistant', + model: 'claude-3-5-sonnet', + content: [{ type: 'text', text: 'Hello there' }], + stop_reason: 'end_turn', + usage: { input_tokens: 12, output_tokens: 5 }, +}; + +describe('anthropic provider', () => { + it('translates an OpenAI request into an Anthropic Messages request', async () => { + const fetchImpl = vi.fn( + async (_url: string, _init: { headers: Record; body: string }) => + Promise.resolve(jsonResponse(reply)), + ); + const provider = createAnthropicProvider({ + name: 'anthropic', + baseUrl: 'http://h/v1/', + apiKey: 'key', + fetchImpl, + }); + + await provider.chat(request); + + const call = fetchImpl.mock.calls[0]!; + expect(call[0]).toBe('http://h/v1/messages'); + expect(call[1].headers['x-api-key']).toBe('key'); + expect(call[1].headers['anthropic-version']).toBe('2023-06-01'); + expect(call[1].headers.authorization).toBeUndefined(); + const body = JSON.parse(call[1].body) as { + model: string; + max_tokens: number; + system?: string; + temperature?: number; + messages: { role: string; content: string }[]; + stream?: boolean; + }; + expect(body.model).toBe('claude-3-5-sonnet'); + expect(body.max_tokens).toBe(100); + expect(body.system).toBe('Be brief.'); // system message hoisted out of messages + expect(body.temperature).toBe(0.5); + expect(body.messages).toEqual([{ role: 'user', content: 'hi' }]); + expect(body.stream).toBeUndefined(); + }); + + it('maps the Anthropic response into the OpenAI chat-completion shape', async () => { + const fetchImpl = vi.fn(() => Promise.resolve(jsonResponse(reply))); + const provider = createAnthropicProvider({ name: 'anthropic', baseUrl: 'http://h', fetchImpl }); + + const result = (await provider.chat(request)) as { + object: string; + model: string; + choices: { message: { role: string; content: string }; finish_reason: string | null }[]; + usage: { prompt_tokens: number; completion_tokens: number; total_tokens: number }; + }; + expect(result.object).toBe('chat.completion'); + expect(result.model).toBe('claude-3-5-sonnet'); + expect(result.choices[0]?.message).toEqual({ role: 'assistant', content: 'Hello there' }); + expect(result.choices[0]?.finish_reason).toBe('stop'); // end_turn → stop + expect(result.usage).toEqual({ prompt_tokens: 12, completion_tokens: 5, total_tokens: 17 }); + }); + + it('defaults max_tokens when the caller omits it, and maps max_tokens → length', async () => { + const fetchImpl = vi.fn( + async (_url: string, _init: { headers: Record; body: string }) => + Promise.resolve(jsonResponse({ ...reply, stop_reason: 'max_tokens' })), + ); + const provider = createAnthropicProvider({ name: 'anthropic', baseUrl: 'http://h', fetchImpl }); + + const result = (await provider.chat({ + model: 'claude-3-5-sonnet', + messages: [{ role: 'user', content: 'hi' }], + })) as { choices: { finish_reason: string | null }[] }; + + const body = JSON.parse(fetchImpl.mock.calls[0]![1].body) as { max_tokens: number }; + expect(body.max_tokens).toBe(4096); + expect(result.choices[0]?.finish_reason).toBe('length'); + }); + + it('omits x-api-key when no key is set', async () => { + const fetchImpl = vi.fn( + async (_url: string, _init: { headers: Record; body: string }) => + Promise.resolve(jsonResponse(reply)), + ); + const provider = createAnthropicProvider({ name: 'anthropic', baseUrl: 'http://h', fetchImpl }); + await provider.chat(request); + expect(fetchImpl.mock.calls[0]![1].headers['x-api-key']).toBeUndefined(); + }); + + it('throws UpstreamError on a non-OK response', async () => { + const fetchImpl = vi.fn(() => Promise.resolve(new Response('overloaded', { status: 529 }))); + const provider = createAnthropicProvider({ name: 'anthropic', baseUrl: 'http://h', fetchImpl }); + await expect(provider.chat(request)).rejects.toBeInstanceOf(UpstreamError); + }); + + it('throws UpstreamError when the response shape is unexpected', async () => { + const fetchImpl = vi.fn(() => Promise.resolve(jsonResponse({ content: 'not-an-array' }))); + const provider = createAnthropicProvider({ name: 'anthropic', baseUrl: 'http://h', fetchImpl }); + await expect(provider.chat(request)).rejects.toBeInstanceOf(UpstreamError); + }); + + it('translates an Anthropic SSE stream into OpenAI chunks', async () => { + const fetchImpl = vi.fn(() => + Promise.resolve( + streamResponse([ + 'event: message_start\ndata: {"type":"message_start","message":{"id":"m"}}\n\n', + 'event: content_block_delta\ndata: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hel"}}\n\n', + 'event: content_block_delta\ndata: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"lo"}}\n\n', + 'event: message_delta\ndata: {"type":"message_delta","delta":{"stop_reason":"end_turn"}}\n\n', + 'event: message_stop\ndata: {"type":"message_stop"}\n\n', + ]), + ), + ); + const provider = createAnthropicProvider({ name: 'anthropic', baseUrl: 'http://h', fetchImpl }); + + const chunks: { choices: { delta: { content?: string }; finish_reason: string | null }[] }[] = + []; + for await (const payload of provider.chatStream(request)) { + chunks.push(JSON.parse(payload)); + } + // message_start and message_stop are dropped; two text deltas + one finish chunk remain. + expect(chunks).toHaveLength(3); + const text = chunks.map((c) => c.choices[0]?.delta.content ?? '').join(''); + expect(text).toBe('Hello'); + expect(chunks[2]?.choices[0]?.finish_reason).toBe('stop'); + }); + + it('throws UpstreamError when a streaming response has no body', async () => { + const fetchImpl = vi.fn(() => Promise.resolve(new Response(null, { status: 200 }))); + const provider = createAnthropicProvider({ name: 'anthropic', baseUrl: 'http://h', fetchImpl }); + await expect( + (async () => { + for await (const _chunk of provider.chatStream(request)) { + // drain + } + })(), + ).rejects.toBeInstanceOf(UpstreamError); + }); + + it('flattens array content, joins system blocks, and forwards top_p / stop', async () => { + const fetchImpl = vi.fn( + async (_url: string, _init: { headers: Record; body: string }) => + Promise.resolve(jsonResponse(reply)), + ); + const provider = createAnthropicProvider({ name: 'anthropic', baseUrl: 'http://h', fetchImpl }); + await provider.chat({ + model: 'claude-3-5-sonnet', + messages: [ + { role: 'system', content: 'A' }, + { role: 'system', content: 'B' }, + { role: 'user', content: [{ type: 'text', text: 'multi' }, { type: 'image' }] }, + { role: 'assistant', content: 'ok' }, + { role: 'tool', content: 'treated-as-user' }, + ], + top_p: 0.9, + stop: ['STOP'], + } as ChatCompletionRequest); + + const body = JSON.parse(fetchImpl.mock.calls[0]![1].body) as { + system: string; + top_p: number; + stop_sequences: string[]; + messages: { role: string; content: string }[]; + }; + expect(body.system).toBe('A\n\nB'); // multiple system blocks joined + expect(body.top_p).toBe(0.9); + expect(body.stop_sequences).toEqual(['STOP']); + expect(body.messages).toEqual([ + { role: 'user', content: 'multi' }, // text block extracted; image block dropped + { role: 'assistant', content: 'ok' }, + { role: 'user', content: 'treated-as-user' }, // non-system/assistant → user + ]); + }); + + it('skips keep-alives, unknown events, and non-text deltas in the stream', async () => { + const fetchImpl = vi.fn(() => + Promise.resolve( + streamResponse([ + ': ping\n\n', // comment, no data line + 'event: ping\ndata: {"type":"ping"}\n\n', // unknown type + 'event: content_block_start\ndata: {"type":"content_block_start","index":0}\n\n', // not a delta + 'event: content_block_delta\ndata: {"type":"content_block_delta","delta":{"type":"input_json_delta"}}\n\n', // non-text delta + 'event: content_block_delta\ndata: not-json\n\n', // unparseable + 'event: content_block_delta\ndata: {"type":"content_block_delta","delta":{"type":"text_delta","text":"hi"}}\n\n', + ]), + ), + ); + const provider = createAnthropicProvider({ name: 'anthropic', baseUrl: 'http://h', fetchImpl }); + const out: string[] = []; + for await (const payload of provider.chatStream(request)) out.push(payload); + expect(out).toHaveLength(1); + const chunk = JSON.parse(out[0]!) as { choices: { delta: { content?: string } }[] }; + expect(chunk.choices[0]?.delta.content).toBe('hi'); + }); +}); diff --git a/packages/gateway/src/providers/anthropic.ts b/packages/gateway/src/providers/anthropic.ts new file mode 100644 index 0000000..70b0e57 --- /dev/null +++ b/packages/gateway/src/providers/anthropic.ts @@ -0,0 +1,274 @@ +import { z } from 'zod'; +import type { ChatCompletionRequest } from '../schemas.js'; +import type { FetchLike, Provider } from './types.js'; +import { UpstreamError } from '../errors.js'; + +/** + * Native adapter for Anthropic's Messages API (`POST /v1/messages`), which is + * NOT OpenAI-compatible: different auth header (`x-api-key`), a top-level + * `system` field, a required `max_tokens`, and a distinct response/stream shape. + * This adapter translates an OpenAI chat request in and an OpenAI response out, + * so the rest of the gateway only ever sees the OpenAI shape. + */ + +const ANTHROPIC_VERSION = '2023-06-01'; +// Anthropic requires max_tokens; OpenAI's is optional. Used when the caller omits it. +const DEFAULT_MAX_TOKENS = 4096; + +export interface AnthropicOptions { + name: string; + baseUrl: string; + apiKey?: string | undefined; + /** Injectable fetch (defaults to global `fetch`); handy for tests. */ + fetchImpl?: FetchLike; +} + +/** The slice of an Anthropic Messages response we translate to the OpenAI shape. */ +const anthropicResponseSchema = z + .object({ + id: z.string().optional(), + model: z.string().optional(), + content: z + .array(z.object({ type: z.string(), text: z.string().optional() }).passthrough()) + .optional(), + stop_reason: z.string().nullable().optional(), + usage: z + .object({ input_tokens: z.number().optional(), output_tokens: z.number().optional() }) + .optional(), + }) + .passthrough(); + +type AnthropicResponse = z.infer; + +/** Creates a provider backed by Anthropic's native Messages API. */ +export function createAnthropicProvider(options: AnthropicOptions): Provider { + const fetchImpl: FetchLike = options.fetchImpl ?? fetch; + const endpoint = `${options.baseUrl.replace(/\/+$/, '')}/messages`; + + function buildHeaders(): Record { + const headers: Record = { + 'content-type': 'application/json', + 'anthropic-version': ANTHROPIC_VERSION, + }; + if (options.apiKey !== undefined && options.apiKey.length > 0) { + headers['x-api-key'] = options.apiKey; + } + return headers; + } + + async function send( + request: ChatCompletionRequest, + stream: boolean, + signal: AbortSignal | undefined, + ): Promise { + const res = await fetchImpl(endpoint, { + method: 'POST', + headers: buildHeaders(), + body: JSON.stringify(toAnthropicBody(request, stream)), + // Don't follow upstream redirects — a malicious provider could 3xx the prompt to another host. + redirect: 'error', + ...(signal ? { signal } : {}), + }); + if (!res.ok) { + throw await UpstreamError.fromResponse(options.name, res); + } + return res; + } + + return { + name: options.name, + async chat(request, signal) { + const res = await send(request, false, signal); + const parsed = anthropicResponseSchema.safeParse(await res.json()); + if (!parsed.success) { + throw new UpstreamError(options.name, 502, 'unexpected Anthropic response shape'); + } + return toOpenAIResponse(parsed.data, request.model); + }, + async *chatStream(request, signal) { + const res = await send(request, true, signal); + if (res.body === null) { + throw new UpstreamError(options.name, 502, 'upstream returned no response body'); + } + yield* translateStream(res.body, request.model); + }, + }; +} + +/** Flattens OpenAI message content (string or text-block array) to plain text. */ +function contentToText(content: unknown): string { + if (typeof content === 'string') return content; + if (Array.isArray(content)) { + return content + .map((part) => + typeof part === 'object' && + part !== null && + typeof (part as { text?: unknown }).text === 'string' + ? (part as { text: string }).text + : '', + ) + .join(''); + } + return ''; +} + +/** Translates an OpenAI chat request into an Anthropic Messages request body. */ +function toAnthropicBody(request: ChatCompletionRequest, stream: boolean): Record { + const systemParts: string[] = []; + const messages: { role: 'user' | 'assistant'; content: string }[] = []; + for (const message of request.messages) { + const text = contentToText(message.content); + if (message.role === 'system') { + if (text.length > 0) systemParts.push(text); + continue; + } + // Anthropic only accepts user/assistant turns; map anything else to user. + messages.push({ role: message.role === 'assistant' ? 'assistant' : 'user', content: text }); + } + + const extra = request as Record; + const topP = typeof extra.top_p === 'number' ? extra.top_p : undefined; + const stop = extra.stop; + const stopSequences = + typeof stop === 'string' + ? [stop] + : Array.isArray(stop) + ? stop.filter((s): s is string => typeof s === 'string') + : undefined; + + const body: Record = { + model: request.model, + max_tokens: request.max_tokens ?? DEFAULT_MAX_TOKENS, + messages, + }; + if (systemParts.length > 0) body.system = systemParts.join('\n\n'); + if (request.temperature !== undefined) body.temperature = request.temperature; + if (topP !== undefined) body.top_p = topP; + if (stopSequences !== undefined && stopSequences.length > 0) body.stop_sequences = stopSequences; + if (stream) body.stream = true; + return body; +} + +/** Builds an OpenAI chat-completion response from an Anthropic Messages response. */ +function toOpenAIResponse(a: AnthropicResponse, requestModel: string): Record { + const text = (a.content ?? []) + .filter((block) => block.type === 'text') + .map((block) => block.text ?? '') + .join(''); + + const response: Record = { + id: a.id ?? 'anthropic', + object: 'chat.completion', + created: Math.floor(Date.now() / 1000), + model: a.model ?? requestModel, + choices: [ + { + index: 0, + message: { role: 'assistant', content: text }, + finish_reason: mapStopReason(a.stop_reason ?? null), + }, + ], + }; + if (a.usage !== undefined) { + const prompt = a.usage.input_tokens ?? 0; + const completion = a.usage.output_tokens ?? 0; + response.usage = { + prompt_tokens: prompt, + completion_tokens: completion, + total_tokens: prompt + completion, + }; + } + return response; +} + +/** Maps an Anthropic `stop_reason` to the OpenAI `finish_reason` vocabulary. */ +function mapStopReason(reason: string | null): string | null { + switch (reason) { + case null: + return null; + case 'max_tokens': + return 'length'; + case 'tool_use': + return 'tool_calls'; + case 'end_turn': + case 'stop_sequence': + default: + return 'stop'; + } +} + +/** Translates an Anthropic SSE stream into OpenAI `chat.completion.chunk` payload strings. */ +async function* translateStream( + body: ReadableStream, + model: string, +): AsyncIterable { + const reader = body.getReader(); + const decoder = new TextDecoder(); + let buffer = ''; + const id = 'anthropic'; + const created = Math.floor(Date.now() / 1000); + try { + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + buffer += decoder.decode(value, { stream: true }); + let separator = buffer.indexOf('\n\n'); + while (separator !== -1) { + const chunk = translateEvent(buffer.slice(0, separator), id, created, model); + buffer = buffer.slice(separator + 2); + if (chunk !== null) yield chunk; + separator = buffer.indexOf('\n\n'); + } + } + } finally { + reader.releaseLock(); + } +} + +/** Translates one Anthropic SSE event into an OpenAI chunk string, or null to skip it. */ +function translateEvent(event: string, id: string, created: number, model: string): string | null { + const dataLine = event.split('\n').find((line) => line.startsWith('data:')); + if (dataLine === undefined) return null; + const payload = dataLine.slice('data:'.length).trim(); + if (payload.length === 0) return null; + + let parsed: unknown; + try { + parsed = JSON.parse(payload); + } catch { + return null; // a keep-alive or non-JSON line — nothing to forward + } + if (typeof parsed !== 'object' || parsed === null) return null; + const obj = parsed as { type?: unknown; delta?: unknown }; + + if (obj.type === 'content_block_delta') { + const delta = obj.delta as { type?: unknown; text?: unknown } | undefined; + if (delta !== undefined && delta.type === 'text_delta' && typeof delta.text === 'string') { + return openAIChunk(id, created, model, { content: delta.text }, null); + } + return null; + } + if (obj.type === 'message_delta') { + const delta = obj.delta as { stop_reason?: unknown } | undefined; + const stop = + delta !== undefined && typeof delta.stop_reason === 'string' ? delta.stop_reason : null; + return openAIChunk(id, created, model, {}, mapStopReason(stop)); + } + return null; +} + +function openAIChunk( + id: string, + created: number, + model: string, + delta: Record, + finishReason: string | null, +): string { + return JSON.stringify({ + id, + object: 'chat.completion.chunk', + created, + model, + choices: [{ index: 0, delta, finish_reason: finishReason }], + }); +} diff --git a/packages/gateway/src/providers/registry.test.ts b/packages/gateway/src/providers/registry.test.ts index 9ea7709..6ead0b8 100644 --- a/packages/gateway/src/providers/registry.test.ts +++ b/packages/gateway/src/providers/registry.test.ts @@ -5,8 +5,24 @@ import type { ResolvedConfig } from '../config.js'; const config: ResolvedConfig = { providers: new Map([ - ['ollama', { name: 'ollama', baseUrl: 'http://localhost:11434/v1', apiKey: undefined }], - ['openai', { name: 'openai', baseUrl: 'https://api.openai.com/v1', apiKey: 'k' }], + [ + 'ollama', + { + name: 'ollama', + type: 'openai-compatible', + baseUrl: 'http://localhost:11434/v1', + apiKey: undefined, + }, + ], + [ + 'openai', + { + name: 'openai', + type: 'openai-compatible', + baseUrl: 'https://api.openai.com/v1', + apiKey: 'k', + }, + ], ]), models: new Map([['qwen2.5:7b', 'ollama']]), defaultProvider: 'openai', @@ -47,4 +63,30 @@ describe('createRegistry', () => { .chat({ model: 'qwen2.5:7b', messages: [{ role: 'user', content: 'hi' }] }); expect(fetchImpl).toHaveBeenCalledOnce(); }); + + it('builds the Anthropic adapter for an anthropic-typed provider', async () => { + const anthropicConfig: ResolvedConfig = { + providers: new Map([ + ['claude', { name: 'claude', type: 'anthropic', baseUrl: 'http://h/v1', apiKey: 'sk-ant' }], + ]), + models: new Map([['claude-3-5-sonnet', 'claude']]), + defaultProvider: undefined, + pricing: new Map(), + }; + const fetchImpl = vi.fn( + async (_url: string, _init: { headers: Record; body: string }) => + Promise.resolve( + new Response(JSON.stringify({ content: [{ type: 'text', text: 'hi' }] }), { + status: 200, + }), + ), + ); + const registry = createRegistry(anthropicConfig, { fetchImpl }); + await registry + .resolve('claude-3-5-sonnet') + .chat({ model: 'claude-3-5-sonnet', messages: [{ role: 'user', content: 'hi' }] }); + const call = fetchImpl.mock.calls[0]!; + expect(call[0]).toContain('/messages'); + expect(call[1].headers['x-api-key']).toBe('sk-ant'); + }); }); diff --git a/packages/gateway/src/providers/registry.ts b/packages/gateway/src/providers/registry.ts index 41659ef..36743bc 100644 --- a/packages/gateway/src/providers/registry.ts +++ b/packages/gateway/src/providers/registry.ts @@ -1,6 +1,7 @@ import type { ResolvedConfig } from '../config.js'; import type { FetchLike, Provider } from './types.js'; import { createOpenAICompatibleProvider } from './openai-compatible.js'; +import { createAnthropicProvider } from './anthropic.js'; import { ModelNotFoundError } from '../errors.js'; export interface ProviderRegistry { @@ -19,14 +20,17 @@ export function createRegistry( ): ProviderRegistry { const providers = new Map(); for (const resolved of config.providers.values()) { + const adapterOptions = { + name: resolved.name, + baseUrl: resolved.baseUrl, + apiKey: resolved.apiKey, + ...(options.fetchImpl ? { fetchImpl: options.fetchImpl } : {}), + }; providers.set( resolved.name, - createOpenAICompatibleProvider({ - name: resolved.name, - baseUrl: resolved.baseUrl, - apiKey: resolved.apiKey, - ...(options.fetchImpl ? { fetchImpl: options.fetchImpl } : {}), - }), + resolved.type === 'anthropic' + ? createAnthropicProvider(adapterOptions) + : createOpenAICompatibleProvider(adapterOptions), ); } diff --git a/sentinel.config.example.json b/sentinel.config.example.json index f910720..647f598 100644 --- a/sentinel.config.example.json +++ b/sentinel.config.example.json @@ -19,6 +19,11 @@ "type": "openai-compatible", "baseUrl": "https://api.groq.com/openai/v1", "apiKeyEnv": "GROQ_API_KEY" + }, + "anthropic": { + "type": "anthropic", + "baseUrl": "https://api.anthropic.com/v1", + "apiKeyEnv": "ANTHROPIC_API_KEY" } }, "models": { @@ -26,7 +31,8 @@ "qwen2.5:0.5b": "ollama", "gpt-4o-mini": "openai", "gemini-2.0-flash": "gemini", - "llama-3.3-70b-versatile": "groq" + "llama-3.3-70b-versatile": "groq", + "claude-3-5-haiku-latest": "anthropic" }, "defaultProvider": "ollama", "routing": { @@ -41,6 +47,7 @@ "pricing": { "gpt-4o-mini": { "inputPer1k": 0.00015, "outputPer1k": 0.0006 }, "gemini-2.0-flash": { "inputPer1k": 0.0001, "outputPer1k": 0.0004 }, - "llama-3.3-70b-versatile": { "inputPer1k": 0.00059, "outputPer1k": 0.00079 } + "llama-3.3-70b-versatile": { "inputPer1k": 0.00059, "outputPer1k": 0.00079 }, + "claude-3-5-haiku-latest": { "inputPer1k": 0.0008, "outputPer1k": 0.004 } } }