Skip to content

Feat/ollama provider#3213

Open
Aneesh-382005 wants to merge 2 commits into
ultraworkers:mainfrom
Aneesh-382005:feat/ollama-provider
Open

Feat/ollama provider#3213
Aneesh-382005 wants to merge 2 commits into
ultraworkers:mainfrom
Aneesh-382005:feat/ollama-provider

Conversation

@Aneesh-382005
Copy link
Copy Markdown

Summary

The codebase already acknowledged this gap in a comment in rust/crates/api/src/providers/mod.rs:

// When OPENAI_BASE_URL is set, the user explicitly configured an
// OpenAI-compatible endpoint. Prefer it over the Anthropic fallback
// even when the model name has no recognized prefix — this is the
// common case for local providers (Ollama, LM Studio, vLLM, etc.)
// where model names like "qwen2.5-coder:7b" don't match any prefix.

The routing logic beneath it was correct. But two layers upstream were
still blocking local model names before routing ever ran.

Before this PR, the intended workaround was:

export OPENAI_BASE_URL="http://127.0.0.1:11434/v1"
export OPENAI_API_KEY="anything"
claw --model "openai/qwen2.5-coder:7b" prompt "say hello"

This required three env vars, a fake API key, and a provider/ prefix
that Ollama doesn't understand, and it still broke due to a logic bug in
wire_model_for_base_url where an inner matches! could never fire,
causing the full openai/qwen2.5-coder:7b slug to reach Ollama's API
instead of the bare model name.

After this PR:

export OLLAMA_HOST="http://127.0.0.1:11434"
claw --model "qwen2.5-coder:7b" prompt "say hello"

One env var. No prefix. No fake key. Matches Ollama's own convention.

What was blocking it

Two separate layers rejected local model names before routing logic ran:

  1. validate_model_syntax() in rusty-claude-cli/src/main.rs enforced
    provider/model format on every model name unconditionally.

  2. detect_provider_kind() in providers/mod.rs matched the qwen
    prefix and routed to DashScope regardless of what endpoint was configured.

What this PR changes

  • rusty-claude-cli/src/main.rs - if OLLAMA_HOST is set, skip the
    provider/model format requirement. Empty strings still rejected.
  • providers/mod.rs - if OLLAMA_HOST is set, skip prefix matching
    and route directly to the OpenAI-compatible backend.
  • providers/openai_compat.rs - adds OLLAMA_CONFIG constant and
    from_ollama_env() constructor that reads OLLAMA_HOST, appends /v1,
    and sets a dummy api_key since Ollama requires no auth.
  • client.rs - in the OpenAi dispatch arm, when OLLAMA_HOST is set,
    use from_ollama_env() instead of the standard constructor.
  • docs/local-openai-compatible-providers.md and USAGE.md updated to
    document OLLAMA_HOST as the preferred Ollama path.

Follow-up work not in this PR

  • ProviderKind::Ollama as a dedicated variant so /doctor shows
    "Ollama" as the provider name rather than "OpenAI"
  • Same pattern for LM Studio and llama.cpp
  • Tool-call graceful degradation for models that don't support
    function calling

Anti-slop triage

  • Classification: actionable-fix
  • Evidence: claw --model "qwen2.5-coder:7b" prompt "x" returns
    invalid model syntax on unpatched main. With this patch and
    OLLAMA_HOST=http://127.0.0.1:11434, qwen2.5-coder:7b responds
    correctly. Regression test added in alias_resolution_tests.
  • Non-destructive review result: merge candidate

Verification

  • Targeted tests/docs checks ran, or the gap is explicitly recorded.
  • git diff --check passes.
  • No live secrets, tokens, private logs, or unrelated generated churn are included.

Resolution gate

  • If this PR resolves an issue, the issue number and fix evidence are linked.
  • If this PR should not merge, the rejection/defer rationale is evidence-backed and does not rely on vibes.
  • I did not merge/close remote PRs or issues from an automation lane without owner approval.

Aneesh-382005 and others added 2 commits June 2, 2026 16:03
Co-authored-by: Copilot <copilot@github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant