A lightweight, Go-based proxy server that provides an Ollama-compatible API while forwarding requests to various LLM backends (OpenAI-compatible APIs like llama.cpp, or actual Ollama instances). All requests and responses are logged to an SQLite database for debugging and analysis.
The main motivation for creating this was to get the Home Assistant Ollama integration to work with llama.cpp. I've also always wanted a better way to log requests and responses from other LLM based apps (i.e. Open Web UI). This proxy has a web interface which shows all of the messages including the full context and the system prompt which is really useful for debugging.
- Ollama-Compatible API - Presents an Ollama API interface, compatible with Home Assistant and other Ollama clients
- Basic OpenAI-Compatible API - Provides
/v1/chat/completionsand/v1/modelsfrontend endpoints for simple OpenAI-style clients - Reasoning Passthrough - Backend reasoning (
reasoning/reasoning_contentfrom OpenAI-compatible servers such as vLLM, or Ollamathinking) is returned asthinkingon Ollama endpoints and asreasoning_contenton/v1/chat/completions - Multiple Backend Support - Connect to OpenAI-compatible APIs (e.g., llama.cpp) or Ollama instances
- Streaming Support - Full support for streaming responses with minimal latency
- Request/Response Logging - All interactions logged to SQLite with timestamps, latency, and error tracking
- Web UI - Built-in interface for viewing logs, request/response details, and configuration
- JSON Logs API - Query logged frontend/backend requests and responses from
/api/logs - Model Metadata Passthrough - Preserves upstream context-window metadata such as
max_model_lenanddetails.context_length - Text Injection - Automatically inject text into user messages (disabled by default) for example "/nothink" to disable thinking
- Tool Blacklist - Filter out specific tools from chat requests before forwarding to the backend
- Request Sanitization - Drop problematic maximum-token parameters from incoming requests
- Docker Support - Production-ready Docker images with health checks
- Minimal Dependencies - Uses Go plus TOML parsing and a pure-Go SQLite driver; no C compiler is required
- Highly Configurable - Fine-tune logging, timeouts, CORS, database cleanup, and more
The easiest way to run LLM Proxy is using the pre-built Docker images:
# 1. Create a config file
curl -O https://raw.githubusercontent.com/stevelittlefish/llm_proxy/master/config.toml.example
mv config.toml.example config.toml
# Edit config.toml to configure your backend
# 2. Create data directory
mkdir -p data
# 3. Run the container
docker run -d \
--name llm-proxy \
-p 11434:11434 \
-v $(pwd)/config.toml:/app/config/config.toml:ro \
-v $(pwd)/data:/app/data \
ghcr.io/stevelittlefish/llm_proxy:latestCreate a docker-compose.yml file:
version: '3.8'
services:
llm-proxy:
image: ghcr.io/stevelittlefish/llm_proxy:latest
container_name: llm-proxy
restart: unless-stopped
ports:
- "11434:11434"
volumes:
- ./config.toml:/app/config/config.toml:ro
- ./data:/app/dataThen run:
# Get the example config
curl -O https://raw.githubusercontent.com/stevelittlefish/llm_proxy/master/config.toml.example
mv config.toml.example config.toml
# Edit config.toml to configure your backend
# Create data directory
mkdir -p data
# Start the service
docker-compose up -dFor advanced Docker setup (building from source, custom networks, etc.), see DOCKER.md.
This proxy is designed to sit between Home Assistant (or any Ollama client) and llama.cpp (or other backends), allowing you to:
- Use Home Assistant's Ollama integration with llama.cpp for faster responses
- Log all LLM interactions for debugging and analysis
- Switch between different backends without reconfiguring clients
- Map model names between different systems
The easiest way to get started is to download a pre-built binary from the releases page.
- Download the Linux x86_64 archive:
llm_proxy_VERSION_linux_amd64.tar.gz
Note: Pre-built binaries are currently only available for Linux x86_64. macOS, Windows, and ARM64 users should build from source or use Docker.
-
Extract the archive:
tar -xzf llm_proxy_VERSION_linux_amd64.tar.gz cd llm_proxy_VERSION_linux_amd64 -
Edit
config.tomlto configure your backend -
Run the proxy:
./llm_proxy
If you prefer to build from source:
- Go 1.25 or later
Clone the repository and then do:
git clone https://github.com/stevelittlefish/llm_proxy
cd llm_proxy
cp config.toml.example config.toml
# Edit the file to change settings
go run .Or use the included helper script:
./run.shgit clone https://github.com/stevelittlefish/llm_proxy
cd llm_proxy
go mod download
CGO_ENABLED=0 go build -o llm_proxySee DOCKER.md for Docker and Docker Compose installation instructions.
Create a config.toml file based on the provided example:
[server]
host = "0.0.0.0"
port = 11434
enable_cors = false
log_messages = true
log_raw_requests = false
log_raw_responses = false
verbose = false
[backend]
type = "openai"
endpoint = "http://localhost:8008"
timeout = 300
tool_blacklist = []
[backend_openai]
force_prompt_cache = false
[database]
path = "./data/llm_proxy.db"
max_requests = 100
cleanup_interval = 5
[request_sanitization]
max_tokens_policy = "preserve"
max_tokens_limit = 0
[chat_text_injection]
enabled = false
text = "/nothink"
mode = "last"
[stream_override]
mode = "passthrough"
[ollama_overrides]
thinking = "passthrough"
[gemma_4_fix]
enabled = falsehost: IP address to bind to (default:0.0.0.0)port: Port to listen on (default:11434- Ollama's default port)enable_cors: Enable CORS middleware (default:false) - this will allow any web page to directly access the server via javascriptlog_messages: Log message content in human-readable format to stdout (default:false)log_raw_requests: Log raw JSON request payloads (pretty-printed) to stdout (default:false)log_raw_responses: Log raw JSON response payloads (pretty-printed) to stdout (default:false)verbose: Enable verbose logging for debugging - logs filtered blacklisted tools, text injection operations, and all HTTP requests/responses with status codes (default:false)
Logging Options:
- All three logging options are independent and can be enabled together
log_messagesprovides human-readable summaries (e.g., "Model: llama2, Prompt: Why is the sky blue?")log_raw_requestsshows the exact JSON payloads received by the proxylog_raw_responsesshows the complete JSON responses (including all streaming chunks)- All logs go to stdout and can be redirected to files if needed
- Note: These are stdout logs only; database logging is always enabled regardless of these settings
type: Backend type - either"openai"or"ollama"endpoint: URL of the backend service- For llama.cpp: typically
http://localhost:8080 - For Ollama: typically
http://localhost:11434
- For llama.cpp: typically
timeout: Request timeout in seconds (default:300)tool_blacklist: List of tool names to filter out from requests (default:[])
Tool Blacklist:
- Filters out specific tools from chat requests before forwarding to the backend
- Only applies to chat endpoints (
/api/chatand/v1/chat/completions; tools are not used in/api/generate) - Useful for blocking unwanted or problematic tools from reaching the LLM
- Tools are matched by their function name
- Filtered tools are logged to stdout for debugging
Example Configuration:
[backend]
tool_blacklist = ["web_search", "execute_code", "sensitive_tool"]Behavior:
- When a chat request includes tools, the proxy checks each tool's function name against the blacklist
- Matching tools are removed from the request before forwarding to the backend
- The LLM will not see or be able to use blacklisted tools
- Filtering happens after text injection but before backend forwarding
- A log message is printed for each filtered tool:
Filtering out blacklisted tool: <name>
force_prompt_cache: Whentrue, automatically addscache_prompt: trueto all OpenAI API requests (default:false)
Prompt Caching:
- Only applies when using OpenAI backend (
"type": "openai") - When enabled, instructs the back-end to cache the prompt for improved performance on repeated queries
- The
cache_promptparameter is automatically injected into both/api/chatand/api/generaterequests - Has no effect when using Ollama backend
path: Path to SQLite database file (default:./data/llm_proxy.db)max_requests: Maximum number of requests to keep in the database (default:100). Older requests are automatically deleted during cleanup.cleanup_interval: How often (in minutes) to run the cleanup task (default:5). Set to0to disable automatic cleanup.
Database Cleanup:
- The cleanup task runs automatically in the background based on the
cleanup_interval - When triggered, it removes the oldest requests, keeping only the most recent
max_requestsentries - The first cleanup runs immediately on startup, then repeats at the configured interval
- Set
max_requeststo0orcleanup_intervalto0to disable automatic cleanup - All request/response data is permanently deleted when cleaned up
max_tokens_policy: How to handle incoming maximum-token parameters (default:"preserve")max_tokens_limit: Threshold used whenmax_tokens_policy = "drop_above"(default:0)
Max Tokens Policy:
- Applies to OpenAI-compatible
max_tokenson/v1/chat/completions - Applies to Ollama-compatible
options.num_predicton/api/chatand/api/generate - Runs after raw request logging, so
log_raw_requests = truestill shows the original client payload - Runs before backend forwarding, so dropped values are omitted from backend requests
Policy Options:
"preserve": Keep incoming values unchanged"drop_above": Drop the parameter only when it is greater thanmax_tokens_limit"drop": Always drop the incoming maximum-token parameter
Example: Drop excessive values only
[request_sanitization]
max_tokens_policy = "drop_above"
max_tokens_limit = 8192Example: Always let the backend choose
[request_sanitization]
max_tokens_policy = "drop"enabled: Enable text injection (default:false)text: The text string to inject (e.g.,"/nothink")mode: Where to inject text -"first","last", or"system"(default:"last")
Text Injection Behavior:
- Disabled by default - must be explicitly enabled in config.toml
- Only applies to chat endpoints (
/api/chatand/v1/chat/completions, not/api/generate) - When enabled, automatically appends the configured text to the specified user message
- Smart injection - checks if the text already exists and skips injection if present
- Text is added with a preceding space:
"hello"becomes"hello /nothink" - Injection happens after raw request logging but before the backend call
- Useful for adding special tokens or instructions to all user messages
Example Configuration:
[chat_text_injection]
enabled = true
text = "/nothink"
mode = "last"Mode Options:
"first": Injects text into the first message withrole == "user"in the messages array"last": Injects text into the last message withrole == "user"in the messages array (typically the current user input)"system": Appends text to the existing system message, or creates one at the start if no system message exists
mode: Force the streaming behavior of the proxy's call to the backend, ignoring what the client asked for (default:"passthrough")
Mode Options:
"passthrough": Ask the backend for whatever the client requested (default, no override)"always": Always ask the backend to stream, even if the client asked for a single non-streamed response"never": Always ask the backend for a single non-streamed response, even if the client asked to stream
Behavior:
- Applies to
/api/chat,/api/generate, and/v1/chat/completions - Only changes how the proxy talks to the backend. The response sent back to the client always matches what the client originally requested: a client that asked to stream still gets an SSE/NDJSON stream (the proxy reshapes one backend response into a single chunk if needed), and a client that asked for a single response still gets exactly one JSON object (the proxy aggregates backend chunks into one if needed)
- Useful when the backend behaves better one way (e.g. avoiding timeouts on long generations by always streaming, or avoiding a backend's broken streaming mode) regardless of what clients request
Example Configuration:
[stream_override]
mode = "always"thinking: Force the Ollama-nativethinkflag on outbound/api/chatrequests (default:"passthrough")
Mode Options:
"passthrough": Do not changethink; preserve a client-supplied value if present"off": Forcethink: falseto disable model reasoning"on": Forcethink: true
Behavior:
- Applies only to chat endpoints (
/api/chatand/v1/chat/completions), not/api/generate - Only materializes when
backend.type = "ollama", because Ollama exposes thinking control on native/api/chat; the OpenAI backend has no equivalent and does not forwardthink - Useful for OpenAI-compatible clients that cannot toggle Ollama thinking through
/v1/chat/completions, while the proxy can still set the native Ollama field on the backend request
Example Configuration:
[ollama_overrides]
thinking = "off"enabled: Enable the Gemma 4 streaming-corruption mitigation (default:false)
Behavior:
- Self-contained, removable mitigation for two known bugs in vLLM's
gemma4tool-call and reasoning parsers, only relevant whenbackend.type = "openai"and the backend is vLLM serving Gemma 4 with--tool-call-parser gemma4 --reasoning-parser gemma4 - Detects when vLLM's tool-call parser leaks raw control-token tool-call syntax into the response
contentfield instead of producing a structuredtool_callsdelta, and transparently retries (or sends a short internal follow-up turn if real content already streamed to the client) so the client never sees the corrupted text - Also strips leaked
<|channel>thought ... <channel|>reasoning-channel wrapper tokens from streamed content - Applies only to the streaming chat path (
/api/chatand/v1/chat/completions); does not affect/api/generateor the Ollama backend - Disabled by default; turning it on does not change behavior for any other model or backend
Example Configuration:
[gemma_4_fix]
enabled = true./llm_proxy -config config.tomlOr use the default config file location:
./llm_proxyWhen running from source, the helper script does the same thing with go run:
./run.shcurl -X POST http://localhost:11434/api/generate \
-H "Content-Type: application/json" \
-d '{
"model": "llama2",
"prompt": "Why is the sky blue?",
"stream": true
}'curl -X POST http://localhost:11434/api/chat \
-H "Content-Type: application/json" \
-d '{
"model": "llama2",
"messages": [
{"role": "user", "content": "Hello, how are you?"}
],
"stream": true
}'curl http://localhost:11434/api/tagsA small dependency-free terminal chat client is included for quick manual testing:
go run ./cmd/chatclient -config config.tomlOr use the helper script:
./client.shBy default it connects to the proxy using the Ollama-compatible API. To use the OpenAI-compatible API instead:
go run ./cmd/chatclient -config config.toml --openaiWith the helper script:
./client.sh --openaiTo test a vision-capable backend through the OpenAI-compatible API, send an image file as the first user message:
./client.sh --openai --image images/example.pngThe client reads the server host and port from the same config file as the proxy. It auto-selects the first model returned by the proxy, or you can choose one explicitly:
go run ./cmd/chatclient -config config.toml --model llama2The client also accepts --url if you want to bypass config-based connection discovery.
Inside the client, use /clear to reset chat history, /model NAME to switch models, and /quit to exit.
In Home Assistant, configure the Ollama integration to point to your proxy:
# configuration.yaml
conversation:
llm:
provider: ollama
base_url: "http://your-proxy-ip:11434"
model: "llama2"The proxy implements the following Ollama API endpoints:
POST /api/generate- Text completionPOST /api/chat- Chat completionGET /api/tags- List available modelsPOST /api/show- Show model information
Model listing and show responses preserve upstream metadata where available, including context length fields used by OpenAI- and Ollama-compatible clients.
If an OpenAI-compatible backend does not provide a usable /v1/models response, model listing falls back to a single model named default so clients that require discovery can still connect.
The proxy implements the following basic OpenAI API endpoints:
POST /v1/chat/completions- Chat completionGET /v1/models- List available models
GET /- Home page with configuration overviewGET /logs- Paginated list of all requests/responsesGET /logs/details?id=<id>- Detailed view of a specific requestGET /logs/download?id=<id>- Download one complete log entry as Markdown for debuggingGET /api/logs- JSON list of logged requests; supports filters such aslimit,offset,model,endpoint,backend_type,status,errors_only,since,until,q,order, andbodiesGET /api/logs/{id}- JSON detail for one logged request, always including frontend/backend request and response bodiesGET /api/logs?id=<id>- Query-parameter form of the same JSON detail endpointGET /health- Health check endpoint (returns "OK")
The web interface provides an easy way to browse logs, inspect request/response details, and monitor the proxy's configuration without needing direct database access. The JSON logs API exposes the same stored request data for debugging tools; see docs/logs-api.md for the full API reference.
Use "type": "openai" to connect to OpenAI-compatible APIs like llama.cpp:
- Translates Ollama requests to OpenAI format
- Converts streaming SSE responses to Ollama's newline-delimited JSON
- Maps parameters (temperature, max_tokens, etc.)
Example llama.cpp command:
./server -m model.gguf --port 8080 --host 0.0.0.0Use "type": "ollama" to wrap an existing Ollama instance:
- Simple pass-through with logging
- Useful for debugging and monitoring Ollama usage
- No translation required
Example Ollama configuration:
[backend]
type = "ollama"
endpoint = "http://localhost:11435"┌─────────────────┐
│ Home Assistant │
│ (Ollama API) │
└────────┬────────┘
│
v
┌─────────────────┐
│ LLM Proxy │
│ - Logging │
│ - Translation │
└────────┬────────┘
│
┌────┴────┐
v v
┌────────┐ ┌────────┐
│ llama │ │ Ollama │
│ .cpp │ │ │
└────────┘ └────────┘
llm_proxy/
├── main.go # Entry point and server setup
├── cmd/
│ └── chatclient/ # Dependency-free terminal chat client
├── config/
│ └── config.go # Configuration loading
├── backend/
│ ├── backend.go # Backend interface
│ ├── openai.go # OpenAI backend implementation
│ ├── gemma4_fix.go # Optional Gemma 4/vLLM stream mitigation
│ └── ollama.go # Ollama backend implementation
├── handlers/
│ ├── generate.go # /api/generate handler
│ ├── chat.go # /api/chat handler
│ ├── models.go # /api/tags and /api/show handlers
│ ├── openai_frontend.go # /v1/chat/completions and /v1/models handlers
│ ├── chat_features.go # Shared chat injection and tool filtering
│ ├── request_sanitization.go # Maximum-token request policies
│ ├── stream_override.go # Backend streaming override
│ ├── logs_api.go # /api/logs JSON handlers
│ ├── web.go # Web UI handlers
│ ├── static/ # Embedded static assets
│ └── templates/ # HTML templates for web UI
│ ├── home.html # Configuration overview
│ ├── logs.html # Request logs list
│ └── details.html # Request details view
├── models/
│ └── types.go # Request/response types
├── database/
│ ├── sqlite.go # SQLite connection and initialization
│ └── queries.go # Database queries
├── middleware/
│ ├── cors.go # CORS middleware
│ └── logging.go # Verbose request logging middleware
├── run.sh # Run the proxy from source
├── client.sh # Run the chat client from source
├── Dockerfile # Docker build configuration
├── docker-compose.yml # Docker compose setup
├── docker-compose.override.yml.example # Local ports and volumes
├── config.toml.example # Example configuration
├── docs/ # Logs API and Gemma 4 mitigation notes
└── README.md # This file
Streaming responses are normally forwarded with minimal latency. The proxy retains response data for SQLite logging, aggregates chunks when a non-streaming client response is required, and may briefly hold suspicious content when the optional Gemma 4 mitigation is enabled.
See DOCKER.md for detailed Docker and Docker Compose instructions.
This project uses automated builds via GitHub Actions and GoReleaser. When you push a tag starting with v, it builds the configured release artifacts and creates a GitHub release.
Use the included make_release.sh script to create and push a release:
./make_release.sh v1.0.0 "Initial release"The script will:
- Validate the version format (must start with
v) - Check for uncommitted changes and warn if found
- Verify you're on the
masterbranch - Check that the tag doesn't already exist
- Show a summary and ask for confirmation
- Create an annotated tag with your message
- Push the tag to GitHub
GitHub Actions will then automatically:
- Build a Linux x86_64 binary
- Build and publish Docker images
- Create release archives with config files and documentation
- Generate checksums
- Create a GitHub release with all artifacts
Manual Release (alternative):
git tag -a v1.0.0 -m "Release v1.0.0"
git push origin v1.0.0Each release includes:
- A pre-built Linux x86_64 binary
config.toml(ready to edit)- Documentation (README.md, LICENCE.md, DOCKER.md)
- Data directory with README
- SHA256 checksums for verification
I didn't write that much of the code, my good friend Claude did through the Cline plugin in Visual Studio Code. It cost me $12.78 in tokens and took around half a day. Not bad! (EDIT: more like $25 now!)
Contributions are welcome! Please feel free to submit issues and pull requests to the repository.
Wow things have moved so quickly with AI!
I built this in Jan 2026 when I hadn't really used proper AI coding tools before. I was running Cline with local models, and swapped it over to Claude to build this. At work I was using a little bit of Copilot but that was it.
Now it's June and I don't write code myself at work any more and rely heavily on Claude Code. I did the latest batch of updates to this project using Codex and it was so quick and easy.
Fingers crossed, in 6 months time I still have a job and my work hasn't been completely automated!