Skip to content
BranchesTags
 
 

Latest commit

 

History

6,818 Commits

Folders and files

NameName
Last commit message
Last commit date
.claude/skills/ollama-preflight
.claude/skills/ollama-preflight
 
 
.github
.github
 
 
anthropic
anthropic
 
 
api
api
 
 
app
app
 
 
auth
auth
 
 
cmake
cmake
 
 
cmd
cmd
 
 
create
create
 
 
discover
discover
 
 
docs
docs
 
 
envconfig
envconfig
 
 
format
format
 
 
fs
fs
 
 
harmony
harmony
 
 
integration
integration
 
 
internal
internal
 
 
llama
llama
 
 
llm
llm
 
 
logutil
logutil
 
 
manifest
manifest
 
 
middleware
middleware
 
 
ml
ml
 
 
mlx
mlx
 
 
mlxrunner
mlxrunner
 
 
model
model
 
 
openai
openai
 
 
parser
parser
 
 
progress
progress
 
 
readline
readline
 
 
scripts
scripts
 
 
server
server
 
 
template
template
 
 
thinking
thinking
 
 
tools
tools
 
 
types
types
 
 
version
version
 
 
x/transfer
x/transfer
 
 
.dockerignore
.dockerignore
 
 
.gitattributes
.gitattributes
 
 
.gitignore
.gitignore
 
 
.golangci.yaml
.golangci.yaml
 
 
AGENTS.md
AGENTS.md
 
 
CLAUDE.md
CLAUDE.md
 
 
CMakeLists.txt
CMakeLists.txt
 
 
CMakePresets.json
CMakePresets.json
 
 
CONTRIBUTING.md
CONTRIBUTING.md
 
 
Dockerfile
Dockerfile
 
 
Dockerfile.applearm
Dockerfile.applearm
 
 
Dockerfile.gemma4budget
Dockerfile.gemma4budget
 
 
Dockerfile.rocm
Dockerfile.rocm
 
 
LICENSE
LICENSE
 
 
LLAMA_CPP_VERSION
LLAMA_CPP_VERSION
 
 
MLX_C_VERSION
MLX_C_VERSION
 
 
MLX_VERSION
MLX_VERSION
 
 
README.md
README.md
 
 
SECURITY.md
SECURITY.md
 
 
go.mod
go.mod
 
 
go.sum
go.sum
 
 
main.go
main.go
 
 

Repository files navigation

This is a fork of ollama/ollama

Everything below this banner is upstream's README, describing upstream's work. Ollama is theirs; this fork only adds to it, and tries to add as little as possible.

It tracks upstream releases and carries a narrow set of changes:

  • Vision-model correctness. Image token budgets and native-aspect dynamic resolution for gemma4 and nemotron, and an fp32-accumulation gate for qwen2.5-vl that closes an fp16 overflow in the vision tower — a few elements in millions reach inf on CUDA and the caption collapses into a repeated glyph. Offered upstream as ollama#18070.
  • A vision regression suite. A preflight harness with recorded per-model expectations, generated (public, no-download) trigger images, and env-gated node-level instrumentation, run against every build before it deploys.
  • Fixes carried until upstream takes them, each tracked against an upstream issue or PR, and deleted from here when it lands there — the retirement register lists every carried item, what retires it, and the test that gates its deletion.
  • An experimental MLX runtime for Apple Silicon and CUDA — see the caveats below before using it for anything that matters.

Current fold: v0.34.1-dynres — upstream v0.34.1, llama.cpp b10864, MLX d9add9d1. main moves ahead of this between folds; the tag is the fixed point to build and roll back to. Deployed: main at 16649e8, stamped 0.34.1-dynres-16-g16649e8, on the CUDA host since 2026-09-18 22:30 — the tag's native payload with a Go-only rebuild that adds ADR 0036 (a gemma4 vision runner starts from the batch rung holding its image ceiling), with OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0 on the container (ADR 0033; see the fold's memory finding in docs/maxusai/tasks/upstream-sync-0.34.1.md). The matrix below is the tag's full preflight run; the rebuild changes no native input, and its verification on production is in the task doc's deploy section. On the Apple Silicon host the same commit has served the mlx-metal surface on :11435 since 2026-09-18, stamped 0.34.0-maxusai-8a7ba949 — the same build as 0.34.1-dynres-0-g8a7ba94 (ADR 0032, 2026-09-19 amendment) — and was promoted on 2026-09-19 with the MLX #3912 kernel fix kept (ADR 0037), with OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0 in its launchd environment as on the CUDA container. The AMD/gfx1151 host joined on 2026-09-19 18:55, stamped 0.34.1-dynres-16649e8c from a full FLAVOR=rocm build of the same commit, when the upgrade gate lifted — it had held that host on 0.32.1 since 2026-07-31, and all three platforms now serve one commit. That build must carry llama/compat/906-revert-hip-integrated-flag.patch: llama.cpp b10864 misses upstream's HIP revert by 78 minutes, and without it vision output on gfx1151 is silently wrong — no crash, no warning, unchanged token counts.

Fork builds are stamped <upstream-version>-dynres-<n>-g<sha>; dynres names the change that started the fork, not the company that runs it. Fork-specific documentation, ADRs and measurements live in docs/maxusai/.

What tested green for this fold

surface Build identity Image size ladder Pinned image budget thinking on/off Output quality fp16 overflow canary Runner isolation measured on
cuda green green green green skipped green green 0.34.1-dynres-0-g8a7ba94
mlx-cuda not run not run not run not run not run not run not run —
mlx-metal skipped green skipped green not run skipped green 0.34.0-maxusai-8a7ba949
apple-silicon-mlx not run not run not run not run not run not run not run —
rocm not run not run not run not run not run not run not run —
cpu not run not run not run not run not run not run not run —

Generated by release_matrix.py from recorded preflight runs. A surface with no run for this release reads not run — absence is shown, never assumed green. A group is reported at its weakest check, so one skipped probe does not read as a pass.

cuda here is the deployed serving surface. apple-silicon-mlx is the deprecated alias for mlx-metal and will disappear from the generator with it.

What differs from upstream, concretely

Measured against upstream ollama v0.34.1 at llama.cpp b10864. Every row is a capability the fork has and upstream does not; the record column is where the decision and its measurements live (docs/maxusai/).

Vision correctness on the llama.cpp path — the deployed engine

upstream ollama this fork record
nemotron-3 vision fixed 512×512 canvas — 256 tokens per image, whatever the aspect ratio native-aspect dynamic resolution, 256–3,328 tokens, position embeddings interpolated to the patch grid in-graph patch 002, ADR 0001
gemma4 image budget default limits 70–1,120 tokens (40–280 before b10864); an under-budget image keeps its natural rounded grid and is letterbox-padded every image scaled to fill the requested budget and snapped to gemma4's supported ladder (70/140/280/560/1120), never padded — off-ladder grids measurably break box_2d vertical grounding. The budget is a per-request option (image_min_tokens/image_max_tokens, defaults 70/1120) and the scheduler reloads when the resolved flags change. Upstream has since adopted the same default limits; the fill is still fork-only patch 004, ADR 0003/0008/0016
qwen2.5-vl on CUDA f16 vision matmuls accumulate in fp16; on some ordinary images a few elements of millions reach inf at v.blk.31.ffn_down and the caption collapses into one repeated glyph fp32 accumulation forced for every qwen25vl runner, keyed on the GGUF architecture. Offered upstream as ollama#18070 llm/llama_server.go
MoE + MMQ on CUDA ids-path tail padding sized from ne11; under broadcast ne11 == 1, so the buffer gets no padding and the kernel overruns by up to a 512-row tile padding sized from the flattened row count. Reported as llama.cpp#27044 patch 903
transparent images pixels as decoded composited over white before the resize, matching the mlx-vlm reference ADR 0015

Structured output and generation control

upstream ollama this fork record
think + format in one request defers the grammar until the thinking→content transition and folds pass-one metrics into the final response the same, plus: a model with a known think-close marker stops pass one exactly there and continues textually, so runaway thinking cannot burn the budget; pass-one metrics are reconstructed when a runner does not report them; the second pass is pinned to pass one's truncation window ADR 0002/0004/0010
drafting under a grammar (MLX) always on on by default to match upstream; OLLAMA_MLX_DRAFT_UNDER_GRAMMAR=0 restores the gate ADR 0033
stop sequences (MLX) not honoured by the MLX runner honoured, with a possible stop prefix held back until it matches or the stream ends mlxrunner/stopper.go
KV cache type one global OLLAMA_KV_CACHE_TYPE per model, with K/V pair syntax and a policy for reasoning models ADR 0005

Serving and scheduling

upstream ollama this fork record
MLX admission weights against free device memory (and, since v0.34.1, a system-memory bound on integrated GPUs) weights + KV priced at the requested num_ctx + a per-architecture headroom; an explicit rung that does not fit is refused, an automatic one is clamped ADR 0034
MLX memory ceiling none OLLAMA_MLX_MEMORY_LIMIT and a cache limit, set per runner from the admitted budget runner knobs
nvfp4 global scales — held in MLX's m × 2688 representation and divided back out by every wrapper that applies the scale itself, which is not the identity in float32 (17 of 31b's 191 vision scales move one ulp) ADR 0039 (proposed)
model identity in a record a tag the manifest digest: the library re-published gemma4:*-nvfp4 with bf16 vision towers under unchanged tags and config blobs ADR 0038 (proposed)
gemma4 image chunk vs. generation batch (GGUF) the batch follows num_ctx (1024 above 4096), so a top-rung gemma4 image (up to 1120 tokens) is decoded in two pieces, bidirectional only within each a gemma4 vision runner starts from the batch rung that holds its image ceiling (2048 at 1120) and steps down only when it does not fit ADR 0036
gemma4 on MLX upstream's own vision and audio tower with a fixed per-checkpoint soft-token set, no per-request budget vision through upstream's MediaModel with a per-request budget seam; audio not shipped ADR 0021
media prompts on MLX — prefill chunks span-aligned around image blocks; a late image is refused ADR 0014
scheduler — log sites never drop fields under contention; head-of-line and evict-all-wait fixes; attached media charged against capabilities before the load; capability advertising corrected for MLX architectures server/sched.go, images.go
panic hygiene (MLX) — a cleanup that fails while a request is unwinding never replaces the panic that caused it mlxrunner/unwind.go

Measurement — nothing comparable upstream

this fork record
vision regression suite preflight with versioned per-model expectations, generated (public, no-download) trigger images, an env-gated node-level meter, and a generated release matrix — run before every deploy ADR 0011/0012, patch 801
campaigns five report templates rendered only by generators; per-request memory and drafting analysis; bbox conformance scoped to image geometry ADR 0012/0028/0030
bounding-box protocol requests pin norm-1000 and carry a self-calibrating anchor, so the model's internal resize cannot contaminate coordinates: 111 of 112 cells convert cleanly across 14 geometries × 4 models × 2 think modes. A protocol and its measurements, not a runtime change ADR 0027/0030
fork identity builds stamped <upstream>-dynres-<n>-g<sha>, a tag per fold, release notes carrying the generated matrix ADR 0032

MLX runtime — experimental, and slower on CUDA

It works. Models load, stay resident and generate correct output on both Metal and CUDA. But it is not the path to reach for by default:

  • On CUDA it runs at 34–75% of the cuda path's decode throughput, median 46%. Measured across four matched model pairs on one host, one server process — full report. "Roughly half" is a fair central estimate and a poor description of any single case: the spread is 2.2× and it is not architectural (the two dense pairs sit at 75% and 53%, the two MoE at 34% and 39%). Two of the four mlx-cuda arms had no stable throughput to quote at all.
  • Its bigger cost is variance, not speed. mlx-cuda's per-request spread is ~5× the cuda path's and reaches 46% within a single arm — same host, same prompt, back to back — while every cuda arm held inside ±1.5% first time. For anything that sets a timeout or compares two builds, that matters more than the ratio. It is per-model: gemma4:31b-nvfp4 reproduced to 1.1% across four independent measurements.
  • On Metal it is the other way round. A matched campaign measured MLX ~2.4× faster than llama-server (gemma4 12b: 121 vs 50 tok/s decode). So the CUDA gap is CUDA-specific, not an MLX property — do not generalise either number to the other platform.
  • Engine and quantization move together in every figure above — nvfp4 on MLX against q4_K_M on GGUF, because those are the artefacts that exist. So the throughput numbers describe the two stacks as shipped, not the engine in isolation, and the same confound makes MLX-vs-GGUF quality an uncontrolled comparison: a quality difference cannot be attributed to the engine either. Nobody has separated them; until someone does, treat "which is better" as open.
  • It is converging with upstream's own MLX work and is expected to be superseded by it; the fork has already retired its constrained-sampling layer in favour of upstream's engine (ADR 0033).

Use GGML/llama-server for anything where throughput or comparability matters.

Every row above is a delta we would rather not have. Each is offered upstream where it is upstream's to take, and deleted from here once it lands there — the qwen25vl gate and the MMQ padding fix are both filed and pending.

ollama

Ollama

Start building with open models.

Download

macOS

curl -fsSL https://ollama.com/install.sh | sh

or download manually

Windows

irm https://ollama.com/install.ps1 | iex

or download manually

Linux

curl -fsSL https://ollama.com/install.sh | sh

Manual install instructions

Docker

The official Ollama Docker image ollama/ollama is available on Docker Hub.

Libraries

Community

Get started

ollama

You'll be prompted to run a model or connect Ollama to your existing agents or applications such as Claude Code, OpenClaw, OpenCode , Codex, Copilot, and more.

Coding

To launch a specific integration:

ollama launch claude

Supported integrations include Claude Code, Codex, Copilot CLI, DeepSeek Harness, Droid, and OpenCode.

AI assistant

Use OpenClaw to turn Ollama into a personal AI assistant across WhatsApp, Telegram, Slack, Discord, and more:

ollama launch openclaw

Chat with a model

Run and chat with Gemma 4:

ollama run gemma4

See ollama.com/library for the full list.

See the quickstart guide for more details.

REST API

Ollama has a REST API for running and managing models.

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [{
    "role": "user",
    "content": "Why is the sky blue?"
  }],
  "stream": false
}'

See the API documentation for all endpoints.

Python

pip install ollama

from ollama import chat

response = chat(model='gemma4', messages=[
  {
    'role': 'user',
    'content': 'Why is the sky blue?',
  },
])
print(response.message.content)

JavaScript

npm i ollama

import ollama from "ollama";

const response = await ollama.chat({
  model: "gemma4",
  messages: [{ role: "user", content: "Why is the sky blue?" }],
});
console.log(response.message.content);

Supported backends

  • llama.cpp project founded by Georgi Gerganov.

Documentation

Community Integrations

Want to add your project? Open a pull request.

Chat Interfaces

Web

Desktop

  • Dify.AI - LLM app development platform
  • AnythingLLM - All-in-one AI app for Mac, Windows, and Linux
  • Maid - Cross-platform mobile and desktop client
  • Witsy - AI desktop app for Mac, Windows, and Linux
  • Cherry Studio - Multi-provider desktop client
  • Ollama App - Multi-platform client for desktop and mobile
  • PyGPT - AI desktop assistant for Linux, Windows, and Mac
  • Alpaca - GTK4 client for Linux and macOS
  • SwiftChat - Cross-platform including iOS, Android, and Apple Vision Pro
  • Enchanted - Native macOS and iOS client
  • RWKV-Runner - Multi-model desktop runner
  • Ollama Grid Search - Evaluate and compare models
  • macai - macOS client for Ollama and ChatGPT
  • AI Studio - Multi-provider desktop IDE
  • Reins - Parameter tuning and reasoning model support
  • ConfiChat - Privacy-focused with optional encryption
  • LLocal.in - Electron desktop client
  • MindMac - AI chat client for Mac
  • Msty - Multi-model desktop client
  • BoltAI for Mac - AI chat client for Mac
  • IntelliBar - AI-powered assistant for macOS
  • Kerlig AI - AI writing assistant for macOS
  • Hillnote - Markdown-first AI workspace
  • Perfect Memory AI - Productivity AI personalized by screen and meeting history

Mobile

SwiftChat, Enchanted, Maid, Ollama App, Reins, and ConfiChat listed above also support mobile platforms.

Code Editors & Development

Libraries & SDKs

Frameworks & Agents

RAG & Knowledge Bases

  • RAGFlow - RAG engine based on deep document understanding
  • R2R - Open-source RAG engine
  • MaxKB - Ready-to-use RAG chatbot
  • Minima - On-premises or fully local RAG
  • Chipper - AI interface with Haystack RAG
  • ARGO - RAG and deep research on Mac/Windows/Linux
  • Archyve - RAG-enabling document library
  • Casibase - AI knowledge base with RAG and SSO
  • BrainSoup - Native client with RAG and multi-agent automation

Bots & Messaging

Terminal & CLI

Productivity & Apps

Observability & Monitoring

  • Opik - Debug, evaluate, and monitor LLM applications
  • OpenLIT - OpenTelemetry-native monitoring for Ollama and GPUs
  • Lunary - LLM observability with analytics and PII masking
  • Langfuse - Open source LLM observability
  • HoneyHive - AI observability and evaluation for agents
  • MLflow Tracing - Open source LLM observability

Database & Embeddings

Infrastructure & Deployment

Cloud

Package Managers

About

A fork of ollama/ollama adding vision-model correctness fixes — image token budgets, dynamic resolution, and an fp16-overflow gate for qwen2.5-vl — plus a vision regression suite. Tracks upstream releases; fixes are offered upstream and dropped here once they land.

ollama.com

Resources

Readme
License

Contributing

Contributing

Security policy

Security policy
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages