Repository navigation
Queued Ollama chat jobs stop failing at 300 s; one job at a time (0.1.88) - #134
Merged
Merged
Conversation
….88) Slow single-GPU providers failed almost every chat job with a bare "fetch failed" at ~300 s (cullen3, GTX 745: 47 of 49 qwen3:4b jobs on 2026-11-02). Two causes: - The poll loop fires every 15 s whether or not the last batch finished, so a node ran several chat jobs at once against one Ollama, which serves one request at a time. The rest waited in its queue. - Ollama sends response headers with the first token, and Node's global fetch (undici) gives up after 300 s without headers. The Ollama backend now streams /api/chat over node:http, which has no headers/body timeout (the job keeps its own expiry and cancel), and the daemon runs at most INFERNET_MAX_CONCURRENT_JOBS chat jobs (default 1); the rest stay assigned and run when a slot frees. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
ThreatCrush Security Scan40 finding(s) HIGH/CRITICAL: 6 | MEDIUM: 14 | LOW: 20
Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Provider
cullen@cullen3:b4ba4a16(GTX 745 4 GB, 2 cores, ~0.7 tok/s) failed 51 of 56 jobs on 2026-11-02, almost all with a barefetch failedat 302-345 s after assignment (queued jobs in one batch failed at 1125/1426/1727 s, exactly 300 s apart).pollJobsfires every 15 s whether or not the previous batch finished, so a node runs several chat jobs at once against one Ollama, which serves one request at a time. The rest wait in its queue.fetch failed.qwen3:4bchat to this node since it is the only one serving it, so the queue never drained.What
/api/chatovernode:http(no headers/body timeout). Job expiry and cancel still apply.INFERNET_MAX_CONCURRENT_JOBSchat jobs (default 1); the rest stay assigned and run when a slot frees.Tests
tests/ollama-http-fetch.test.jsruns a fake Ollama with a delayed first byte, abort while queued, and cancel. Full suite: 972 passed.🤖 Generated with Claude Code