Skip to content

cloudflare-containers.md

Latest commit

 

History

History
305 lines (226 loc) · 22.6 KB

cloudflare-containers.md

File metadata and controls

305 lines (226 loc) · 22.6 KB
Raw

Deploying MultiLLM-Proxy to Cloudflare Containers

This repo uses a hybrid Cloudflare Worker plus Container deployment. The Worker serves health checks, roleplay session orchestration, native LinkAPI traffic, Codex Everywhere OpenAI traffic, and the Kimi Code model catalog directly. OpenCode provider traffic, Kimi Code chat, and the remaining Flask routes run through a Container.

Why Containers, not Python Workers

Cloudflare's Python Workers runtime runs on Pyodide. Cloudflare's docs note that threading, multiprocessing, and sockets are not functional there. This app uses:

  • Flask and WSGI-style request handling
  • requests for upstream provider calls
  • threaded execution and SSE streaming
  • service-account JSON support for the googleai provider

That makes Cloudflare Containers the safer target for the Python application. /v1/roleplay, /linkapi/*, and /codex-easy/* keep their orchestration in the Worker. The roleplay route uses one SQLite-backed Durable Object per session and sends only selected OpenCode provider calls through Container egress. The Worker serves /kimi-code/v1/models locally and rejects unauthorized or invalid Kimi requests without waking the Container; valid Kimi chat streams through Flask because Kimi's edge rejects Worker-origin egress.

Prerequisites

The main Worker now binds the private multillm-knowledge service. Before its first deployment with that binding, provision and deploy the Knowledge Worker. Knowledge starts disabled; provider secrets and allowances are configured separately from chat.

  1. The target Cloudflare account must have the Workers Paid plan because Containers are only available there.
  2. Authenticate Wrangler against the student roll-number account:
npx wrangler login
  1. Install the JavaScript dependencies used by the Worker:
npm ci

Secrets and variables

Wrangler vars in wrangler.jsonc already set:

  • FLASK_ENV=production
  • SERVER_HOST=0.0.0.0
  • SERVER_PORT=8080
  • APP_NAME=MultiLLM Proxy

RUNTIME_FEATURE_FLAGS groups eight nonsecret settings into one JSON binding to leave room for provider secrets. Worker vars and secrets share a limit of 128 text bindings, and Cloudflare rejects an upload that exceeds it. The Container, usage ledger, roleplay settings and roleplay session read these grouped settings. An individual binding with the same setting name takes precedence, including an explicit disabled value. Keep credentials in separate Worker secrets; the grouped binding accepts only the ten names listed in worker/runtime-feature-flags.mjs. Add a new nonsecret setting to the group rather than as a separate var. It can hold the model price table (MODEL_PRICING_USD_PER_MILLION); keep the group under Cloudflare's 5 KB limit for one variable.

Public HTTP requests receive a 308 redirect to the same HTTPS URL before any route or container work. For Container requests, the Worker overwrites X-Forwarded-Proto, X-Forwarded-Host, and the internal X-MultiLLM-External-Origin header, then enables MULTILLM_TRUST_PROXY_HEADERS inside the Container. The internal origin header survives Container runtime header normalization, so Flask generates public HTTPS dashboard, documentation, and login URLs without trusting caller-supplied forwarding headers in direct local deployments.

Set secrets for anything sensitive:

npx wrangler secret put ADMIN_API_KEY
npx wrangler secret put FLASK_SECRET_KEY
npx wrangler secret put JWT_SECRET
npx wrangler secret put CODEX_EASY_API_KEY
npx wrangler secret put KIMI_CODE_API_KEY
npx wrangler secret put LINKAPI_KEY
npx wrangler secret put OPENCODE_GO_API_KEY
npx wrangler secret put NANOGPT_API_KEY
npx wrangler secret put NANOGPT_API_KEY_1
npx wrangler secret put NAVYAI_API_KEY
npx wrangler secret put AIHUBMIX_API_KEY
npx wrangler secret put GEMINI_API_KEY
npx wrangler secret put OPENAI_API_KEY
npx wrangler secret put OPENROUTER_API_KEY

Set only the providers you actually use. The Worker forwards these directly into the container. It also forwards numbered GROQ_API_KEY_N, NANOGPT_API_KEY_N, and compatibility NANO_GPT_KEY_N secrets automatically. The committed Worker variables set NANOGPT_BILLING_MODE=subscription and NANOGPT_SUBSCRIPTION_BASE_URL=https://nano-gpt.com/api/subscription. These are non-secret routing settings; keep the API keys in Wrangler secrets. Do not add an X-Billing-Mode: paygo override when the NanoGPT account is subscription only. NANOGPT_PREFERRED_KEY_INDEX=0 makes the primary NANOGPT_API_KEY the first choice while retaining numbered credentials as automatic fallbacks.

Native roleplay sessions

POST /v1/roleplay and /roleplay/v1/chat/completions authenticate at the edge and route directly to ROLEPLAY_SESSION, a SQLite-backed Durable Object binding. One object per session serializes turns without creating a global bottleneck. It stores bounded continuity memory, idempotency guards, and Kimi/GLM latency and reliability statistics.

NanoGPT subscription GLM-5.3-Flash is the first production GLM choice, followed by uncensored Flash and GLM-5.2 in the 1x pool. Full GLM-5.3 remains an explicit 2x selector. roleplay:glm-speed ranks the 1x choices using same-session streaming TPS; OpenCode and NavyAI remain later provider tiers. LinkAPI is Kimi-only and OpenRouter is omitted from roleplay. See the roleplay guide for the request schema, compaction policy, safe fallback boundary, metrics route, and all ROLEPLAY_* variables.

NanoGPT and NavyAI routes are Container-backed raw gateways. The Worker forwards their keys, optional base-URL overrides, provider request limits, and the credential-splitting CORS headers into Flask. See NanoGPT gateway and NavyAI gateway.

CODEX_EASY_API_KEY is the preferred Codex Everywhere secret; CODEX_API_KEY remains a fallback alias. Both the Worker and Flask always use the fixed upstream origin https://codex-easy.ai.

Codex Everywhere OpenAI routes

Use $PROXY_BASE_URL/codex-easy when the client appends /v1 itself. Use $PROXY_BASE_URL/codex-easy/v1 when the client expects its configured base URL to include /v1 already.

Operation Direct Worker route Caller authentication
Key-group model catalog /codex-easy/v1/models Authorization: Bearer $ADMIN_API_KEY
OpenAI Responses /codex-easy/v1/responses Authorization: Bearer $ADMIN_API_KEY
Chat Completions /codex-easy/v1/chat/completions Authorization: Bearer $ADMIN_API_KEY
Images /codex-easy/v1/images/* Authorization: Bearer $ADMIN_API_KEY

The Worker authenticates direct callers only against ADMIN_API_KEY, removes that caller credential, and sends CODEX_EASY_API_KEY or its alias upstream. These routes bypass Flask dashboard-user authentication, application-level request-size checks, RPM/TPM/daily limits, Flask request/rate-limit accounting, and request metrics. Use the Container-backed /v1/responses or /v1/chat/completions route with a codex-easy:<model> model ID when those controls are required.

The model catalog is API-key-group-specific; query /codex-easy/v1/models instead of relying on a hard-coded catalog. /v1/images/* works only for image-generation key groups. Raw request and response bytes are preserved for JSON, SSE, binary images, and multipart bodies, except that a JSON GPT Image generation missing moderation is normalized once to insert "low".

On the Codex Everywhere and LinkAPI raw OpenAI fast paths, a Responses prompt_cache_key remains in the untouched body and the Chat X-Grok-Conv-Id header is forwarded. For Grok requests, xAI recommends those fields with a stable conversation ID to improve cache routing. They do not guarantee a cache hit; caching remains upstream behavior and stable request prefixes still matter. Generation POSTs are single-attempt, and this proxy does not provide idempotency.

Codex Everywhere pools

A Codex Everywhere key belongs to one group, so each pool is its own provider with its own Worker secret. All pools use https://codex-everywhere.com. Model IDs take the pool's prefix, such as ce-gpt-pro:gpt-6.1-sol.

Provider Secret Models Price
ce-gpt-plus CODEX_EVERYWHERE_API_KEY_GPT_PLUS_POOL GPT-6 Astra, 6.1 Sol, Sol, Luna; GPT-5.6 0.03× OpenAI
ce-gpt-pro CODEX_EVERYWHERE_API_KEY_GPT_PRO_POOL Same as Plus 0.05× OpenAI
ce-grok-heavy CODEX_EVERYWHERE_API_KEY_GROK_HEAVY Grok 4.7, 4.6, 4.5 0.06× xAI
ce-claude-kiro-cheap CODEX_EVERYWHERE_API_KEY_CLAUDE_UNSTABLE Claude Opus 5.5 and earlier 0.045× Anthropic
ce-claude-kiro CODEX_EVERYWHERE_API_KEY_CLAUDE_KIRO Claude Opus 5.5 and earlier 0.08× Anthropic
ce-claude-max CODEX_EVERYWHERE_API_KEY_CLAUDE_MAX Claude Fable 5.1 and the rest; Claude Code only 0.24× Anthropic
ce-image CODEX_EVERYWHERE_API_KEY_GPT_IMAGE GPT Image 2 and 2.5 Sunburst and Flare $0.015 per image

The GPT and Grok pools accept Chat Completions and Responses, and can be auto:intelligence candidates. The Claude pools accept Anthropic Messages; unified /v1/chat/completions requests to them are translated to Messages, and they are not intelligence candidates. ce-image is an image relay. Raw routes such as /ce-gpt-pro/v1/responses and /ce-claude-kiro/v1/messages accept only each protocol's documented paths, and each pool's /v1/models lists its own key's catalog. A pool with no secret is skipped. CE answers 403 INSUFFICIENT_BALANCE while the account has no credit.

Each pool can hold spare keys under the same name with _1, _2 and so on, such as CODEX_EVERYWHERE_API_KEY_GPT_PRO_POOL_1. Requests use the base key first, then the spares in number order. A key that CE refuses (401, 402 or 403) rests for five minutes, and a rate-limited key (429) for one minute. In auto:intelligence a refused key retries the same model on the next key within the request, because a refusal generates nothing. Other routes switch on the next request. The rests are kept per Container process.

CE's GPT API runs on Codex and uses Codex's default system prompt unless a request sets one (CE docs). Check how a pool treats a system message before adding it to a chain.

The Gemini via Antigravity group serves Google's native API at /v1beta and is not integrated.

Kimi Code OpenAI-compatible routes

Set KIMI_CODE_API_KEY as a Worker secret. The upstream base is fixed to https://api.kimi.com/coding/v1.

Operation Proxy route Execution path Caller authentication
Model catalog /kimi-code/v1/models Worker-local configured catalog Authorization: Bearer $ADMIN_API_KEY
Chat Completions /kimi-code/v1/chat/completions Edge auth, then Container raw pass-through Authorization: Bearer $ADMIN_API_KEY

Kimi Code generation is Chat Completions only in this integration. Use model k3 on the raw route. Send reasoning_effort: "max" for K3's strongest reasoning setting, and keep prompt_cache_key stable for stable conversation prefixes when seeking upstream cache affinity. Cache hits are not guaranteed.

The Worker authenticates the caller with ADMIN_API_KEY before forwarding valid chat requests to Flask. Flask replaces the caller credential with KIMI_CODE_API_KEY and preserves the OpenAI-compatible request/response stream. The raw route bypasses unified request-size checks, rate limits, and accounting; use /v1/chat/completions with model kimi-code:k3 when those controls are required. Chat generation is single-attempt to avoid duplicated work and billing.

LINKAPI_KEY is used both by the direct Worker fast path and the Container fallback. LINKAPI_API_KEY is supported as an alias, but LINKAPI_KEY is preferred. The optional, non-secret LINKAPI_BASE_URL variable defaults to LinkAPI's HK direct origin, https://hk.linkapi.ai, and is restricted to the Worker's allowlist of official LinkAPI hosts; arbitrary HTTPS origins are rejected.

LinkAPI native routes

Use the deployed Worker URL as the base and keep the native path after /linkapi:

Protocol Route Caller authentication
Claude Messages /linkapi/v1/messages x-api-key: $ADMIN_API_KEY and anthropic-version
OpenAI Responses /linkapi/v1/responses Authorization: Bearer $ADMIN_API_KEY
OpenAI compatible /linkapi/v1/chat/completions Authorization: Bearer $ADMIN_API_KEY
OpenAI model catalog /linkapi/v1/models Authorization: Bearer $ADMIN_API_KEY
OpenAI image generation /linkapi/v1/images/generations Authorization: Bearer $ADMIN_API_KEY
OpenAI image editing /linkapi/v1/images/edits Authorization: Bearer $ADMIN_API_KEY
Gemini native /linkapi/v1beta/models/{model}:generateContent Prefer x-goog-api-key: $ADMIN_API_KEY; ?key=$ADMIN_API_KEY is compatibility-only

These direct routes authenticate only with ADMIN_API_KEY. They intentionally bypass Flask dashboard-user authentication, application-level request-size checks, RPM/TPM/daily limits, Flask request/rate-limit accounting, and request metrics. The Worker removes the caller credential and sends LINKAPI_KEY upstream. When the Flask controls are required, use the Container-backed /v1/chat/completions endpoint with a linkapi:<model> model ID, or /v1/images/generations with an image-capable linkapi:<model> model ID.

Gemini clients should prefer x-goog-api-key. Query-string ?key= remains available for compatibility, but it places the caller key in the URL, where clients and intermediaries may retain it, even though automatic Worker invocation logs are disabled.

Request and response bodies are streamed without changing native SSE event frames. A JSON GPT Image generation missing moderation is normalized once to insert "low"; explicit auto and low values remain unchanged. The raw OpenAI routes preserve Responses prompt_cache_key and Chat X-Grok-Conv-Id; for Grok, xAI recommends those shapes for cache routing, but neither this proxy nor LinkAPI guarantees a cache hit. Generation POSTs are single-attempt: the proxy never retries them and does not provide idempotency, because a retry can duplicate upstream work and billing. A caller should retry only when the selected upstream protocol and endpoint explicitly document an idempotency guarantee.

LinkAPI's live pricing page lists gpt-image-2-c for /linkapi/v1/images/generations and /linkapi/v1/images/edits. Use native Gemini generateContent for Gemini Flash Image models such as gemini-2.5-flash-image. See the LinkAPI image guide for request examples and live-catalog guidance.

AIHubMix free models and image routes

Set AIHUBMIX_API_KEY as a Worker secret. The committed non-secret variables use https://aihubmix.com first and https://api.inferera.com as the trusted backup. The Worker forwards all /aihubmix/* routes to the Container so Flask can enforce the path allowlist, preserve multipart image edits, normalize Gemini and Doubao image responses, and record provider metrics.

Operation Route
Live provider catalog /aihubmix/v1/models
Native chat /aihubmix/v1/chat/completions
Native GPT image generation /aihubmix/v1/images/generations
Native GPT image edit /aihubmix/v1/images/edits
Unified GPT, Gemini, or Doubao image generation /v1/images/generations with aihubmix:<model>

See the AIHubMix guide for the seeded free catalog, examples, and the bounded backup-origin policy.

Adaptive and explicit context optimization

The Container-backed normal /v1/chat/completions route safely preprocesses GLM-5.2 context above GLM_AUTO_OPTIMIZE_TRIGGER_TOKENS (8,000 by default). It only replaces older high-confidence image-prompt blocks, preserving surrounding story text, the newest prompt, protected directives, recent turns, media, tools, and reasoning structures. It makes no extra provider call and never summarizes ordinary text. /v1/responses and provider-specific routes remain unchanged.

Prompt classification uses a bounded process-local cache keyed by SHA-256. Values contain prompt-span offsets only, not conversation text or responses. Configure CONTEXT_ANALYSIS_CACHE_ENABLED, CONTEXT_ANALYSIS_CACHE_TTL_SECONDS, and CONTEXT_ANALYSIS_CACHE_MAX_ENTRIES; each Container keeps its own cache.

POST /optimize/v1/chat/completions remains the explicit route for every unified model and uses the unified namespace, authentication, request limits, rate limits, accounting, and metrics.

The default deterministic mode does not call another model. Once the request exceeds optimization.trigger_input_tokens, it can replace older high-confidence detailed image prompts while retaining the newest detailed image prompt and safety-critical structures such as system/developer messages, recent turns, media exchanges, tool chains, and reasoning/thinking blocks. target_input_tokens defaults to 75% of the selected provider's configured prompt limit; both token counts are provider-neutral byte-based estimates rather than exact tokenizer or billing usage.

summarize mode requires optimization.summary_model in provider:model form. If safe older plain-text history must be condensed, the route makes exactly one additional billed and separately rate-limited summary call. One-attempt summary transport is limited to codex-easy, kimi-code, and linkapi; calls use a bounded 45-second read timeout and two-slot per-process pool, and failure or local saturation falls back without retry. The response body and SSE bytes remain upstream-native, while X-MultiLLM-Optimization*, X-MultiLLM-Estimated-Input-*, X-MultiLLM-Image-Prompts-Compacted, X-MultiLLM-Messages-Summarized, X-MultiLLM-Optimization-Cache-Hits, X-MultiLLM-Optimization-Cache-Misses, and X-MultiLLM-Summary headers report what happened and are exposed through CORS.

Summary models are same-provider by default. Eligible historical user/assistant plaintext is transmitted verbatim to the selected summary provider, so it may contain sensitive text. Prefer the same provider, remove secrets, and use preserve_message_indices for history that must stay out of the summary. Cross-provider transfer requires the explicit optimization.allow_cross_provider_summary: true disclosure opt-in; otherwise the request returns 400. The digest is inserted as an untrusted historical assistant message.

OPTIMIZER_MAX_REQUEST_BYTES controls the pre-parse optimizer ingress cap (16 MiB by default), while the transformed final body must still fit the selected provider's MAX_REQUEST_BYTES. OPTIMIZER_SUMMARY_TIMEOUT_SECONDS defaults to 45 seconds and is clamped to 5-120 seconds. The target model/key, output cap, and RPM/daily capacity are checked before a paid summary.

Gemini vs Vertex-style Google auth

If you only use the Gemini/Gemma provider paths, set:

npx wrangler secret put GEMINI_API_KEY

If you also use the googleai provider that depends on Google Cloud auth, set:

npx wrangler secret put GOOGLE_APPLICATION_CREDENTIALS_JSON
npx wrangler secret put PROJECT_ID
npx wrangler secret put LOCATION
npx wrangler secret put GOOGLE_ENDPOINT

GOOGLE_APPLICATION_CREDENTIALS_JSON should contain the full service account JSON as a single secret value. The container entrypoint writes it to /tmp/google-credentials.json and sets GOOGLE_APPLICATION_CREDENTIALS. The app uses that JSON directly through google-auth; gcloud is only a local fallback when no service-account JSON/path is configured.

Deploy

Pushes to main deploy through GitHub Actions once ci passes: D1 migrations first, then the Knowledge Worker, then this Worker and its Container. See How deploys work for the pipeline, the one-time token and secret setup, and the choice between it and Workers Builds.

For a manual deploy, use the ordered script, never a bare wrangler deploy:

npm run deploy            # scripts/deploy.sh

It applies intelligence-migrations/ to the multillm-intelligence D1 database, stops unless every migration file is recorded as applied, then deploys the Knowledge Worker (wrangler.knowledge.jsonc) and finally this Worker and its Container. Extra arguments go to the final deploy, for example npm run deploy -- --containers-rollout immediate for a one-shot rollout of a changed Container image. /ready answers 503 with d1_schema_missing and the missing tables until the schema is migrated; npm run cf:status shows it with the deployed versions and pending migrations.

Alerts. Worker logs are sampled at 100% (head_sampling_rate: 1): the Worker writes little besides structured failure lines such as account_storage_failed, knowledge_edge_account_lookup_failed and d1_schema_missing, about 50,000 events a month at current traffic. In Observability, save queries on those events and on HTTP 5xx for multillm-proxy and multillm-knowledge, and create a notification (or a monitor on /ready) so an outage pages someone instead of being found by users.

The final wrangler deploy builds the container image from Dockerfile, pushes it to Cloudflare, and deploys the Worker, Durable Objects and container binding.

Runtime shape

  • Worker entry: cloudflare-worker.mjs
  • Direct Worker route: /codex-easy/*
  • Direct Worker route: /kimi-code/*
  • Direct Worker route: /linkapi/*
  • Direct Worker route: /v1/roleplay
  • Durable Object class: RoleplaySession
  • Durable Object / Container class: MultiLLMProxyContainer
  • Direct Worker route: signed /v1/media/files/* links, served from R2
  • Workflow class: MediaJobWorkflow, and the optional MEDIA_BUCKET and MEDIA_JOBS bindings (media storage)
  • Container port: 8080
  • Gunicorn entrypoint: app:create_app()
  • Gunicorn default: GUNICORN_WORKERS=1, GUNICORN_THREADS=8

Notes

  • The deployment is pinned to a single named container instance (primary), and wrangler.jsonc sets max_instances=1. Control-plane state is shared through D1, so more instances are possible; see what running more than one requires.
  • Container disk is ephemeral. Dashboard accounts (with AUTH_STORAGE_BACKEND=d1), automatic routes, rate-limit usage, login throttling, model-disable overrides, free-route cooldowns, workbench records and provider catalog snapshots are stored in D1 (details) and survive Container restarts. Keep ADMIN_API_KEY as the bootstrap credential; it authenticates even while D1 is unavailable.