Skip to content
stevelittlefishPublic

About

LLM logging and translation proxy - exposes an ollama + openai compatible API and proxies requests onto either an Ollama server or an OpenAI compatible server (i.e. llama.cpp). Logs all messages and allows easy integration of Home Assistant with llama.cpp.

Resources

Stars

10 stars

Watchers

0 watching

Forks

Latest commit

 

History

116 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

LLM Proxy

A lightweight, Go-based proxy server that provides an Ollama-compatible API while forwarding requests to various LLM backends (OpenAI-compatible APIs like llama.cpp, or actual Ollama instances). All requests and responses are logged to an SQLite database for debugging and analysis.

The main motivation for creating this was to get the Home Assistant Ollama integration to work with llama.cpp. I've also always wanted a better way to log requests and responses from other LLM based apps (i.e. Open Web UI). This proxy has a web interface which shows all of the messages including the full context and the system prompt which is really useful for debugging.

Features

  • Ollama-Compatible API - Presents an Ollama API interface, compatible with Home Assistant and other Ollama clients
  • Basic OpenAI-Compatible API - Provides /v1/chat/completions and /v1/models frontend endpoints for simple OpenAI-style clients
  • Reasoning Passthrough - Backend reasoning (reasoning/reasoning_content from OpenAI-compatible servers such as vLLM, or Ollama thinking) is returned as thinking on Ollama endpoints and as reasoning_content on /v1/chat/completions
  • Multiple Backend Support - Connect to OpenAI-compatible APIs (e.g., llama.cpp) or Ollama instances
  • Streaming Support - Full support for streaming responses with minimal latency
  • Request/Response Logging - All interactions logged to SQLite with timestamps, latency, and error tracking
  • Web UI - Built-in interface for viewing logs, request/response details, and configuration
  • JSON Logs API - Query logged frontend/backend requests and responses from /api/logs
  • Model Metadata Passthrough - Preserves upstream context-window metadata such as max_model_len and details.context_length
  • Text Injection - Automatically inject text into user messages (disabled by default) for example "/nothink" to disable thinking
  • Tool Blacklist - Filter out specific tools from chat requests before forwarding to the backend
  • Request Sanitization - Drop problematic maximum-token parameters from incoming requests
  • Docker Support - Production-ready Docker images with health checks
  • Minimal Dependencies - Uses Go plus TOML parsing and a pure-Go SQLite driver; no C compiler is required
  • Highly Configurable - Fine-tune logging, timeouts, CORS, database cleanup, and more

Quick Start with Docker

The easiest way to run LLM Proxy is using the pre-built Docker images:

Using Docker Run

# 1. Create a config file
curl -O https://raw.githubusercontent.com/stevelittlefish/llm_proxy/master/config.toml.example
mv config.toml.example config.toml
# Edit config.toml to configure your backend

# 2. Create data directory
mkdir -p data

# 3. Run the container
docker run -d \
  --name llm-proxy \
  -p 11434:11434 \
  -v $(pwd)/config.toml:/app/config/config.toml:ro \
  -v $(pwd)/data:/app/data \
  ghcr.io/stevelittlefish/llm_proxy:latest

Using Docker Compose

Create a docker-compose.yml file:

version: '3.8'

services:
  llm-proxy:
    image: ghcr.io/stevelittlefish/llm_proxy:latest
    container_name: llm-proxy
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ./config.toml:/app/config/config.toml:ro
      - ./data:/app/data

Then run:

# Get the example config
curl -O https://raw.githubusercontent.com/stevelittlefish/llm_proxy/master/config.toml.example
mv config.toml.example config.toml
# Edit config.toml to configure your backend

# Create data directory
mkdir -p data

# Start the service
docker-compose up -d

For advanced Docker setup (building from source, custom networks, etc.), see DOCKER.md.

Use Case

This proxy is designed to sit between Home Assistant (or any Ollama client) and llama.cpp (or other backends), allowing you to:

  • Use Home Assistant's Ollama integration with llama.cpp for faster responses
  • Log all LLM interactions for debugging and analysis
  • Switch between different backends without reconfiguring clients
  • Map model names between different systems

Installation

Download Pre-built Binaries

The easiest way to get started is to download a pre-built binary from the releases page.

  1. Download the Linux x86_64 archive: llm_proxy_VERSION_linux_amd64.tar.gz

Note: Pre-built binaries are currently only available for Linux x86_64. macOS, Windows, and ARM64 users should build from source or use Docker.

  1. Extract the archive:

    tar -xzf llm_proxy_VERSION_linux_amd64.tar.gz
    cd llm_proxy_VERSION_linux_amd64
  2. Edit config.toml to configure your backend

  3. Run the proxy:

    ./llm_proxy

Build from Source

If you prefer to build from source:

Prerequisites

  • Go 1.25 or later

Quickstart

Clone the repository and then do:

git clone https://github.com/stevelittlefish/llm_proxy
cd llm_proxy
cp config.toml.example config.toml
# Edit the file to change settings
go run .

Or use the included helper script:

./run.sh

Build Binary

git clone https://github.com/stevelittlefish/llm_proxy
cd llm_proxy
go mod download
CGO_ENABLED=0 go build -o llm_proxy

Docker Installation

See DOCKER.md for Docker and Docker Compose installation instructions.

Configuration

Create a config.toml file based on the provided example:

[server]
host = "0.0.0.0"
port = 11434
enable_cors = false
log_messages = true
log_raw_requests = false
log_raw_responses = false
verbose = false

[backend]
type = "openai"
endpoint = "http://localhost:8008"
timeout = 300
tool_blacklist = []

[backend_openai]
force_prompt_cache = false

[database]
path = "./data/llm_proxy.db"
max_requests = 100
cleanup_interval = 5

[request_sanitization]
max_tokens_policy = "preserve"
max_tokens_limit = 0

[chat_text_injection]
enabled = false
text = "/nothink"
mode = "last"

[stream_override]
mode = "passthrough"

[ollama_overrides]
thinking = "passthrough"

[gemma_4_fix]
enabled = false

Configuration Options

Server

  • host: IP address to bind to (default: 0.0.0.0)
  • port: Port to listen on (default: 11434 - Ollama's default port)
  • enable_cors: Enable CORS middleware (default: false) - this will allow any web page to directly access the server via javascript
  • log_messages: Log message content in human-readable format to stdout (default: false)
  • log_raw_requests: Log raw JSON request payloads (pretty-printed) to stdout (default: false)
  • log_raw_responses: Log raw JSON response payloads (pretty-printed) to stdout (default: false)
  • verbose: Enable verbose logging for debugging - logs filtered blacklisted tools, text injection operations, and all HTTP requests/responses with status codes (default: false)

Logging Options:

  • All three logging options are independent and can be enabled together
  • log_messages provides human-readable summaries (e.g., "Model: llama2, Prompt: Why is the sky blue?")
  • log_raw_requests shows the exact JSON payloads received by the proxy
  • log_raw_responses shows the complete JSON responses (including all streaming chunks)
  • All logs go to stdout and can be redirected to files if needed
  • Note: These are stdout logs only; database logging is always enabled regardless of these settings

Backend

  • type: Backend type - either "openai" or "ollama"
  • endpoint: URL of the backend service
    • For llama.cpp: typically http://localhost:8080
    • For Ollama: typically http://localhost:11434
  • timeout: Request timeout in seconds (default: 300)
  • tool_blacklist: List of tool names to filter out from requests (default: [])

Tool Blacklist:

  • Filters out specific tools from chat requests before forwarding to the backend
  • Only applies to chat endpoints (/api/chat and /v1/chat/completions; tools are not used in /api/generate)
  • Useful for blocking unwanted or problematic tools from reaching the LLM
  • Tools are matched by their function name
  • Filtered tools are logged to stdout for debugging

Example Configuration:

[backend]
tool_blacklist = ["web_search", "execute_code", "sensitive_tool"]

Behavior:

  • When a chat request includes tools, the proxy checks each tool's function name against the blacklist
  • Matching tools are removed from the request before forwarding to the backend
  • The LLM will not see or be able to use blacklisted tools
  • Filtering happens after text injection but before backend forwarding
  • A log message is printed for each filtered tool: Filtering out blacklisted tool: <name>

Backend OpenAI

  • force_prompt_cache: When true, automatically adds cache_prompt: true to all OpenAI API requests (default: false)

Prompt Caching:

  • Only applies when using OpenAI backend ("type": "openai")
  • When enabled, instructs the back-end to cache the prompt for improved performance on repeated queries
  • The cache_prompt parameter is automatically injected into both /api/chat and /api/generate requests
  • Has no effect when using Ollama backend

Database

  • path: Path to SQLite database file (default: ./data/llm_proxy.db)
  • max_requests: Maximum number of requests to keep in the database (default: 100). Older requests are automatically deleted during cleanup.
  • cleanup_interval: How often (in minutes) to run the cleanup task (default: 5). Set to 0 to disable automatic cleanup.

Database Cleanup:

  • The cleanup task runs automatically in the background based on the cleanup_interval
  • When triggered, it removes the oldest requests, keeping only the most recent max_requests entries
  • The first cleanup runs immediately on startup, then repeats at the configured interval
  • Set max_requests to 0 or cleanup_interval to 0 to disable automatic cleanup
  • All request/response data is permanently deleted when cleaned up

Request Sanitization

  • max_tokens_policy: How to handle incoming maximum-token parameters (default: "preserve")
  • max_tokens_limit: Threshold used when max_tokens_policy = "drop_above" (default: 0)

Max Tokens Policy:

  • Applies to OpenAI-compatible max_tokens on /v1/chat/completions
  • Applies to Ollama-compatible options.num_predict on /api/chat and /api/generate
  • Runs after raw request logging, so log_raw_requests = true still shows the original client payload
  • Runs before backend forwarding, so dropped values are omitted from backend requests

Policy Options:

  • "preserve": Keep incoming values unchanged
  • "drop_above": Drop the parameter only when it is greater than max_tokens_limit
  • "drop": Always drop the incoming maximum-token parameter

Example: Drop excessive values only

[request_sanitization]
max_tokens_policy = "drop_above"
max_tokens_limit = 8192

Example: Always let the backend choose

[request_sanitization]
max_tokens_policy = "drop"

Chat Text Injection

  • enabled: Enable text injection (default: false)
  • text: The text string to inject (e.g., "/nothink")
  • mode: Where to inject text - "first", "last", or "system" (default: "last")

Text Injection Behavior:

  • Disabled by default - must be explicitly enabled in config.toml
  • Only applies to chat endpoints (/api/chat and /v1/chat/completions, not /api/generate)
  • When enabled, automatically appends the configured text to the specified user message
  • Smart injection - checks if the text already exists and skips injection if present
  • Text is added with a preceding space: "hello" becomes "hello /nothink"
  • Injection happens after raw request logging but before the backend call
  • Useful for adding special tokens or instructions to all user messages

Example Configuration:

[chat_text_injection]
enabled = true
text = "/nothink"
mode = "last"

Mode Options:

  • "first": Injects text into the first message with role == "user" in the messages array
  • "last": Injects text into the last message with role == "user" in the messages array (typically the current user input)
  • "system": Appends text to the existing system message, or creates one at the start if no system message exists

Stream Override

  • mode: Force the streaming behavior of the proxy's call to the backend, ignoring what the client asked for (default: "passthrough")

Mode Options:

  • "passthrough": Ask the backend for whatever the client requested (default, no override)
  • "always": Always ask the backend to stream, even if the client asked for a single non-streamed response
  • "never": Always ask the backend for a single non-streamed response, even if the client asked to stream

Behavior:

  • Applies to /api/chat, /api/generate, and /v1/chat/completions
  • Only changes how the proxy talks to the backend. The response sent back to the client always matches what the client originally requested: a client that asked to stream still gets an SSE/NDJSON stream (the proxy reshapes one backend response into a single chunk if needed), and a client that asked for a single response still gets exactly one JSON object (the proxy aggregates backend chunks into one if needed)
  • Useful when the backend behaves better one way (e.g. avoiding timeouts on long generations by always streaming, or avoiding a backend's broken streaming mode) regardless of what clients request

Example Configuration:

[stream_override]
mode = "always"

Ollama Overrides

  • thinking: Force the Ollama-native think flag on outbound /api/chat requests (default: "passthrough")

Mode Options:

  • "passthrough": Do not change think; preserve a client-supplied value if present
  • "off": Force think: false to disable model reasoning
  • "on": Force think: true

Behavior:

  • Applies only to chat endpoints (/api/chat and /v1/chat/completions), not /api/generate
  • Only materializes when backend.type = "ollama", because Ollama exposes thinking control on native /api/chat; the OpenAI backend has no equivalent and does not forward think
  • Useful for OpenAI-compatible clients that cannot toggle Ollama thinking through /v1/chat/completions, while the proxy can still set the native Ollama field on the backend request

Example Configuration:

[ollama_overrides]
thinking = "off"

Gemma 4 Fix

  • enabled: Enable the Gemma 4 streaming-corruption mitigation (default: false)

Behavior:

  • Self-contained, removable mitigation for two known bugs in vLLM's gemma4 tool-call and reasoning parsers, only relevant when backend.type = "openai" and the backend is vLLM serving Gemma 4 with --tool-call-parser gemma4 --reasoning-parser gemma4
  • Detects when vLLM's tool-call parser leaks raw control-token tool-call syntax into the response content field instead of producing a structured tool_calls delta, and transparently retries (or sends a short internal follow-up turn if real content already streamed to the client) so the client never sees the corrupted text
  • Also strips leaked <|channel>thought ... <channel|> reasoning-channel wrapper tokens from streamed content
  • Applies only to the streaming chat path (/api/chat and /v1/chat/completions); does not affect /api/generate or the Ollama backend
  • Disabled by default; turning it on does not change behavior for any other model or backend

Example Configuration:

[gemma_4_fix]
enabled = true

Usage

Start the Server

./llm_proxy -config config.toml

Or use the default config file location:

./llm_proxy

When running from source, the helper script does the same thing with go run:

./run.sh

Test with curl

Generate Endpoint (Streaming)

curl -X POST http://localhost:11434/api/generate \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama2",
    "prompt": "Why is the sky blue?",
    "stream": true
  }'

Chat Endpoint (Streaming)

curl -X POST http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama2",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ],
    "stream": true
  }'

List Models

curl http://localhost:11434/api/tags

Chat Client

A small dependency-free terminal chat client is included for quick manual testing:

go run ./cmd/chatclient -config config.toml

Or use the helper script:

./client.sh

By default it connects to the proxy using the Ollama-compatible API. To use the OpenAI-compatible API instead:

go run ./cmd/chatclient -config config.toml --openai

With the helper script:

./client.sh --openai

To test a vision-capable backend through the OpenAI-compatible API, send an image file as the first user message:

./client.sh --openai --image images/example.png

The client reads the server host and port from the same config file as the proxy. It auto-selects the first model returned by the proxy, or you can choose one explicitly:

go run ./cmd/chatclient -config config.toml --model llama2

The client also accepts --url if you want to bypass config-based connection discovery.

Inside the client, use /clear to reset chat history, /model NAME to switch models, and /quit to exit.

Configure Home Assistant

In Home Assistant, configure the Ollama integration to point to your proxy:

# configuration.yaml
conversation:
  llm:
    provider: ollama
    base_url: "http://your-proxy-ip:11434"
    model: "llama2"

API Endpoints

Ollama-Compatible Endpoints

The proxy implements the following Ollama API endpoints:

  • POST /api/generate - Text completion
  • POST /api/chat - Chat completion
  • GET /api/tags - List available models
  • POST /api/show - Show model information

Model listing and show responses preserve upstream metadata where available, including context length fields used by OpenAI- and Ollama-compatible clients. If an OpenAI-compatible backend does not provide a usable /v1/models response, model listing falls back to a single model named default so clients that require discovery can still connect.

OpenAI-Compatible Endpoints

The proxy implements the following basic OpenAI API endpoints:

  • POST /v1/chat/completions - Chat completion
  • GET /v1/models - List available models

Web UI Endpoints

  • GET / - Home page with configuration overview
  • GET /logs - Paginated list of all requests/responses
  • GET /logs/details?id=<id> - Detailed view of a specific request
  • GET /logs/download?id=<id> - Download one complete log entry as Markdown for debugging
  • GET /api/logs - JSON list of logged requests; supports filters such as limit, offset, model, endpoint, backend_type, status, errors_only, since, until, q, order, and bodies
  • GET /api/logs/{id} - JSON detail for one logged request, always including frontend/backend request and response bodies
  • GET /api/logs?id=<id> - Query-parameter form of the same JSON detail endpoint
  • GET /health - Health check endpoint (returns "OK")

The web interface provides an easy way to browse logs, inspect request/response details, and monitor the proxy's configuration without needing direct database access. The JSON logs API exposes the same stored request data for debugging tools; see docs/logs-api.md for the full API reference.

Backend Types

OpenAI Backend

Use "type": "openai" to connect to OpenAI-compatible APIs like llama.cpp:

  • Translates Ollama requests to OpenAI format
  • Converts streaming SSE responses to Ollama's newline-delimited JSON
  • Maps parameters (temperature, max_tokens, etc.)

Example llama.cpp command:

./server -m model.gguf --port 8080 --host 0.0.0.0

Ollama Backend

Use "type": "ollama" to wrap an existing Ollama instance:

  • Simple pass-through with logging
  • Useful for debugging and monitoring Ollama usage
  • No translation required

Example Ollama configuration:

[backend]
type = "ollama"
endpoint = "http://localhost:11435"

Architecture

┌─────────────────┐
│ Home Assistant  │
│  (Ollama API)   │
└────────┬────────┘
         │
         v
┌─────────────────┐
│   LLM Proxy     │
│ - Logging       │
│ - Translation   │
└────────┬────────┘
         │
    ┌────┴────┐
    v         v
┌────────┐  ┌────────┐
│ llama  │  │ Ollama │
│  .cpp  │  │        │
└────────┘  └────────┘

Project Structure

llm_proxy/
├── main.go                 # Entry point and server setup
├── cmd/
│   └── chatclient/         # Dependency-free terminal chat client
├── config/
│   └── config.go           # Configuration loading
├── backend/
│   ├── backend.go          # Backend interface
│   ├── openai.go           # OpenAI backend implementation
│   ├── gemma4_fix.go       # Optional Gemma 4/vLLM stream mitigation
│   └── ollama.go           # Ollama backend implementation
├── handlers/
│   ├── generate.go         # /api/generate handler
│   ├── chat.go             # /api/chat handler
│   ├── models.go           # /api/tags and /api/show handlers
│   ├── openai_frontend.go  # /v1/chat/completions and /v1/models handlers
│   ├── chat_features.go    # Shared chat injection and tool filtering
│   ├── request_sanitization.go # Maximum-token request policies
│   ├── stream_override.go  # Backend streaming override
│   ├── logs_api.go         # /api/logs JSON handlers
│   ├── web.go              # Web UI handlers
│   ├── static/             # Embedded static assets
│   └── templates/          # HTML templates for web UI
│       ├── home.html       # Configuration overview
│       ├── logs.html       # Request logs list
│       └── details.html    # Request details view
├── models/
│   └── types.go            # Request/response types
├── database/
│   ├── sqlite.go           # SQLite connection and initialization
│   └── queries.go          # Database queries
├── middleware/
│   ├── cors.go             # CORS middleware
│   └── logging.go          # Verbose request logging middleware
├── run.sh                  # Run the proxy from source
├── client.sh               # Run the chat client from source
├── Dockerfile              # Docker build configuration
├── docker-compose.yml      # Docker compose setup
├── docker-compose.override.yml.example # Local ports and volumes
├── config.toml.example     # Example configuration
├── docs/                   # Logs API and Gemma 4 mitigation notes
└── README.md               # This file

Performance

Streaming responses are normally forwarded with minimal latency. The proxy retains response data for SQLite logging, aggregates chunks when a non-streaming client response is required, and may briefly hold suspicious content when the optional Gemma 4 mitigation is enabled.

Docker Deployment

See DOCKER.md for detailed Docker and Docker Compose instructions.

Releases

This project uses automated builds via GitHub Actions and GoReleaser. When you push a tag starting with v, it builds the configured release artifacts and creates a GitHub release.

Creating a Release

Use the included make_release.sh script to create and push a release:

./make_release.sh v1.0.0 "Initial release"

The script will:

  1. Validate the version format (must start with v)
  2. Check for uncommitted changes and warn if found
  3. Verify you're on the master branch
  4. Check that the tag doesn't already exist
  5. Show a summary and ask for confirmation
  6. Create an annotated tag with your message
  7. Push the tag to GitHub

GitHub Actions will then automatically:

  1. Build a Linux x86_64 binary
  2. Build and publish Docker images
  3. Create release archives with config files and documentation
  4. Generate checksums
  5. Create a GitHub release with all artifacts

Manual Release (alternative):

git tag -a v1.0.0 -m "Release v1.0.0"
git push origin v1.0.0

Release Artifacts

Each release includes:

  • A pre-built Linux x86_64 binary
  • config.toml (ready to edit)
  • Documentation (README.md, LICENCE.md, DOCKER.md)
  • Data directory with README
  • SHA256 checksums for verification

Contributing

I didn't write that much of the code, my good friend Claude did through the Cline plugin in Visual Studio Code. It cost me $12.78 in tokens and took around half a day. Not bad! (EDIT: more like $25 now!)

Contributions are welcome! Please feel free to submit issues and pull requests to the repository.

Note: 2026-06-19

Wow things have moved so quickly with AI!

I built this in Jan 2026 when I hadn't really used proper AI coding tools before. I was running Cline with local models, and swapped it over to Claude to build this. At work I was using a little bit of Copilot but that was it.

Now it's June and I don't write code myself at work any more and rely heavily on Claude Code. I did the latest batch of updates to this project using Codex and it was so quick and easy.

Fingers crossed, in 6 months time I still have a job and my work hasn't been completely automated!

About

LLM logging and translation proxy - exposes an ollama + openai compatible API and proxies requests onto either an Ollama server or an OpenAI compatible server (i.e. llama.cpp). Logs all messages and allows easy integration of Home Assistant with llama.cpp.

Resources

Stars

10 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages