feat(deepseek4): run the vision encoder on a second GPU and stream images into prefill - #758
Merged
Merged
Conversation
…ages into prefill --mmproj-device hip:N loads the DS4V encoder on another GPU in the one-GPU layout (the R9700 next to a Strix Halo holding the model). Its scratch is charged to that GPU. The encoder then runs image by image on a background thread and publishes each image's rows; prefill waits per chunk only for the images that chunk contains, so the Strix Halo prefills image k while the R9700 encodes image k+1. Failure or cancellation releases waiters and the thread is joined before the request, shutdown or park returns. Requests may carry up to 16 images (one shared constant for the HTTP transport, DS4V and Qwen3.5), and each request logs its encode time. lucebox6, Strix Halo decoder, published DSpark launch, ChartQA charts, time to first token, Strix encoder -> R9700 encoder streamed: 1 image 2.97 -> 2.94 s, 4 images 11.2 -> 9.1 s, 8 images 27.7 -> 19.2 s, 16 images (4,358 tokens) 51.8 -> 34.6 s. Answers identical to the sequential encode. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A throw in the --mmproj-device encoder thread would terminate the server; it now fails the stream so prefill stops waiting. Encode-time logs print only on success, and <thread> joins the system includes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
davide221
added a commit
that referenced
this pull request
Sep 24, 2026
* feat(qwen35): speculative decoding for image requests Image tokens take 2D rotary positions, so after an image the rotary position runs rope_delta_ ahead of the KV position. AR decode already applied that offset; the DFlash verify target did not, so the HTTP layer and the backend forced every image request to plain AR decode. Qwen35DFlashTarget now reads the backend's per-request rope_delta_ and shifts the M-RoPE positions of chain and tree verify by it (zero for text, so text requests are unchanged). The blanket AR force for images is dropped from the HTTP layer and the Qwen3.5 backend; DeepSeek4 keeps its own image guard and still decodes image requests AR. R9700, Qwen3.8-27B-IQ4_XS-pure + Q8_0 projector + DFlash2, 12 images, 256-token answers: 4.03 s per answer (76 tok/s) vs 7.58 s (36 tok/s) before; llama.cpp with the same drafter 5.54 s, without 8.36 s. Image eval 188/220 unchanged (218 answers identical), 1-4 images 9/9. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(deepseek4): DSpark speculative decoding for image requests The image prefill graph takes no DSpark capture hooks, so image requests skipped feature capture and decoded AR. Image chunks now end at their last image, the text after the image prefills and captures as ordinary chunks, and the feature window is cleared at each image chunk so the drafter always reads one contiguous tail. With that, image requests take the DSpark path like text. Strix Halo alone, Vision-Exp ROCMFPX MIX, published DSpark launch, 12 images, 256-token answers: 13.3 s per answer (30.4 tok/s) vs 15.7 s (22.1 tok/s); first token about 0.7 s later from the capture band. 220 image questions 181 (AI2D 86, ChartQA 55/40), 209 identical to plain decode; one to four images 9/9; text decode unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(deepseek4): run the vision encoder on a second GPU and stream images into prefill --mmproj-device hip:N loads the DS4V encoder on another GPU in the one-GPU layout (the R9700 next to a Strix Halo holding the model). Its scratch is charged to that GPU. The encoder then runs image by image on a background thread and publishes each image's rows; prefill waits per chunk only for the images that chunk contains, so the Strix Halo prefills image k while the R9700 encodes image k+1. Failure or cancellation releases waiters and the thread is joined before the request, shutdown or park returns. Requests may carry up to 16 images (one shared constant for the HTTP transport, DS4V and Qwen3.5), and each request logs its encode time. lucebox6, Strix Halo decoder, published DSpark launch, ChartQA charts, time to first token, Strix encoder -> R9700 encoder streamed: 1 image 2.97 -> 2.94 s, 4 images 11.2 -> 9.1 s, 8 images 27.7 -> 19.2 s, 16 images (4,358 tokens) 51.8 -> 34.6 s. Answers identical to the sequential encode. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(qwen35): serve image requests in the concurrent batch A vision projector used to switch concurrent sequence scheduling off for image input. The batched engine now accepts image requests: - SeqEngine gains supports_images() and admit_images(); the default refuses. The scheduler routes image requests there and never gives them a prefix-cache plan (tokens alone do not identify an image). - Qwen35SeqEngine encodes the images at admission, keeps the payload, rows and rope offset with the slot until it retires, overwrites image rows and writes the image's M-RoPE positions in every prefill chunk that covers an image (so an eviction re-prefill sees them again), and shifts decode and chain-verify rotary positions by the slot's offset. - The server enables image input when the engine supports it; the feature gate allows --mmproj with --max-concurrency for Qwen3.5. DeepSeek4 still requires one request at a time. lucebox6 R9700, Qwen3.8-27B-IQ4_XS-pure + Q8_0 projector + DFlash2, --paged-attention --max-concurrency 4: sanity 6/6, one to four images 9/9 (same as single-request); 1/2/4 concurrent 256-token image answers 81/104/149 tok/s total (single-request server 78/73/77), 4 answers in 6.9 s instead of 13.3 s, each answer about its own chart. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(deepseek4): serve image requests in the concurrent batch DS4V image blocks need whole-block bidirectional prefill; the batched engine's gathered step is 16 causal rows. So an image request admitted to the batch is prefilled up to its last token on the single-request sparse path into a staging cache (the encoder streams from --mmproj-device when set), that state is copied into the request's paged slot, and the last token prefills in the batch, producing the first token through the normal step. Decode then runs alongside every other sequence. - import_deepseek4_paged_slot copies the 128-row raw ring, the completed compressed and indexer rows through the slot's block table, and the compressor states, checking layouts and row counts. - DeepSeek4SeqEngine::admit_images seeds the slot with seed_restored_prefix and retires it on any failure. - do_prefill can stop after a prefix and takes its attention mode from the cache it fills (identical to the config for the single-request cache). - Paged serving with --mmproj creates the staging cache (sparse) and lets one image request per slot through the image gate; the feature gate allows DeepSeek4 --mmproj with --paged-attention batching. lucebox6, Strix Halo decoder, R9700 encoder, 4 slots: sanity 2/2, one to four images 9/9; 4 concurrent 256-token image answers in 39.1 s (26.2 tok/s total; 1/2 at once: 17.4/21.7 tok/s), 2 images + 2 texts in 33.1 s (30.9 tok/s), 4 texts 37.9 tok/s. Stacked on #758, #754 and #759. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * perf(deepseek4): prefill concurrent image requests in one shared pass Image admissions now only encode their images and seed their slot; the next batched step prefills every pending image request together. deepseek4_prefill_multi runs one layer-major pass over several sequences: attention per sequence against its own staging cache (with its image masks), HC mixing and the MoE FFN once over all rows, so each layer's expert weights are read once for every request in the pass. The states are then copied into the paged slots as before. A failed request fails only its own slot. lucebox6, Strix Halo + R9700 encoder, 4 slots: 4 image requests share one 1,020-row pass (7.1 s); 4 concurrent 256-token image answers 39.1 -> 35.2 s (26.2 -> 29.1 tok/s), 2+2 mixed 30.9 -> 31.5 tok/s, sanity 2/2, one to four images 9/9. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(vision): address review on image speculative decode - DeepSeek4 falls back to plain decode when an image request has no text after the last image to seed the drafter window. - vision::last_image_end_in replaces the inline span loop in do_prefill, with unit checks. - Clarify the http_server comment and the image-input doc figures. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(qwen35): claim the slot before encoding a batched image request A busy pool defers the request and retries it; encoding first reran the vision tower on every retry. A failed encode now retires the slot. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(deepseek4): keep exceptions inside the image stream thread A throw in the --mmproj-device encoder thread would terminate the server; it now fails the stream so prefill stops waiting. Encode-time logs print only on success, and <thread> joins the system includes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(deepseek4): claim the slot before encoding a batched image request A busy pool defers the request and retries it; encoding first reran the encoder on the scheduler thread on every retry. Names the layer-major prefill minimum (DS4_MIN_LAYER_MAJOR_PREFILL_TOKENS) instead of a bare 5. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(vision): answer 503 when the image request gate is full With batched DS4V serving, image requests beyond one per slot were refused with HTTP 400, which clients do not retry. prepare_images now returns an ImagePrepareStatus (ok, invalid, busy) and the server maps busy to 503. Adds a gate capacity test and brings the DS4V batching docs up to the shared staged prefill (4 image answers 35 s, 29 tok/s). lucebox6: 6 concurrent image requests on 4 slots -> 4 answered in 35.1 s, 2 x 503; test_server_unit 595/595, DS4V image unit tests pass. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(deepseek4): queue image requests instead of refusing them The image request gate refused every image request beyond one per slot (one per backend without batching). A waiting request holds only its preprocessed patches, a few MB per image, and its encoded rows exist only once it runs, so the slots already bound them: DeepSeek image requests now wait in the scheduler queue like text and Qwen image requests. The gate and its lease are removed; short host memory answers 503 (busy). lucebox6: batched, 6 image requests on 4 slots all answered (4 at 35.1 s, 2 queued at 59.6 s), one encode each; single-request server, 2 at once both answered (14.5 s, 30.4 s); sanity 2/2; test_server_unit 595/595. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: mrciffa <davide@lucebox.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
On a box with an R9700 next to a Strix Halo, DeepSeek V4 Flash Vision can run its image encoder on the R9700 while the model stays on the Strix Halo. The encoder streams each image into prefill as soon as it's encoded. Requests may now carry up to 16 images.
Changes
--mmproj-device hip:Nloads the DS4V encoder on another GPU. It applies only in the one-GPU layout; with experts split across two GPUs the encoder already runs on the R9700. The encoder's 2 GiB of scratch is charged to that GPU instead of the target.Results
lucebox6, Strix Halo decoder, the published DSpark launch, ChartQA charts. Time to first token:
Encode time alone, Strix Halo against R9700: 3.9 → 1.9 s for 4 images, 19.0 → 9.3 s for 16.
🤖 Generated with Claude Code