feat(vision): speculative decoding for image requests (Qwen3.8 and DeepSeek V4 Vision)

docs/image-input.md

@@ -36,8 +36,8 @@ hf download Lucebox/Qwen3.8-27B-DFlash2-GGUF \
   --port 8216
 ```
 
-About 21 GiB of VRAM at the peak of an image request. Text requests keep the
-DFlash2 drafter; image requests decode without it.
+About 21 GiB of VRAM at the peak of an image request. Text and image requests
+both decode with the DFlash2 drafter.
 
 ### DeepSeek V4 Flash Vision on a Strix Halo
 
@@ -106,10 +106,10 @@ Decoder pixel and aspect limits also apply. A model's image marker cannot be sup
 as ordinary text.
 
 The server expands image markers after final rendering and tokenization, and
-the expanded image tokens count toward context and usage. Image requests use
-plain autoregressive decoding and bypass the token-keyed prefix, disk and
-agent-turn caches and prompt compression: tokens alone do not identify an
-image. Text requests on the same server keep speculative decoding and caching.
+the expanded image tokens count toward context and usage. Image requests
+bypass the token-keyed prefix, disk and agent-turn caches and prompt
+compression: tokens alone do not identify an image. Image requests decode
+with the model's drafter like text requests.
 
 Layer or tensor splitting across GPUs, remote target shards, concurrent
 sequence scheduling (`--max-concurrency`) and upstream forwarding do not
@@ -144,15 +144,21 @@ Lucebox `Qwen3.8-27B-IQ4_XS-pure` file and a Q8_0 projector:
   lmms-eval prompts: AI2D 90/100, ChartQA relaxed accuracy 56/60 (augmented)
   and 42/60 (human). Image prompts prefill in 0.56 s on average; one to four
   images per request all answer correctly (four images, 2,495 tokens: 3.2 s).
-- Text decodes at 56 to 117 tok/s on 256-token answers (84 on average); image
-  requests decode without the drafter at about 36 tok/s.
+- Text decodes at 56 to 117 tok/s on 256-token answers (84 on average).
+- Image requests decode with the drafter. On 12 images with 256-token
+  answers: 4.0 s per answer (76 tok/s after the first token), against 5.5 s
+  for llama.cpp with the same drafter (`--spec-type draft-dflash`) and 8.4 s
+  without one; faster than llama.cpp with the drafter on every image, 1.21x
+  to 1.58x. The 220-question score
+  is unchanged (188, 218 answers identical to plain decode).
 
 With unsloth's UD-IQ4_XS file and the published BF16 projector:
 
 - 220 seeded questions from `lmms-lab/ai2d` and `lmms-lab/ChartQA` with
   lmms-eval prompts: AI2D 85/100, ChartQA relaxed accuracy 55/60 (augmented)
   and 43/60 (human), no errors. Image prompts average 448 tokens and prefill in
-  0.71 s (largest 1,068 tokens, 1.8 s); decode runs at 31 to 35 tok/s.
+  0.71 s (largest 1,068 tokens, 1.8 s); decode runs at 31 to 35 tok/s (plain
+  decode, measured before image requests used the drafter).
 - A projector with its weight matrices in Q8_0 (rows that are not a multiple
   of 32 stay F16) encodes a 975-token image in 443 ms instead of 677 ms with
   the BF16 file, with the same scores on the 220 questions and 216 identical
@@ -167,7 +173,8 @@ With unsloth's UD-IQ4_XS file and the published BF16 projector:
   tokens). The projector adds 0.9 GiB of VRAM; the peak during image requests
   was 21.6 GiB against 20.8 GiB for text.
 - The same requests answer correctly on a Strix Halo alone, where a
-  1,012-token image prompt prefills in 4.6 s and decodes at 14 tok/s.
+  1,012-token image prompt prefills in 4.6 s and decodes at 14 tok/s (plain
+  decode).
 
 Not yet established: a comparison against the reference implementation on the
 same questions, and CUDA. The tower uses only standard ggml
@@ -246,7 +253,8 @@ the exported projector, on a Strix Halo alone and on R9700 + Strix Halo, 220
 seeded questions from `lmms-lab/ai2d` and `lmms-lab/ChartQA` with lmms-eval
 prompts: AI2D 85/100, ChartQA relaxed accuracy 55/60 (augmented) and 43/60
 (human). Both layouts score the same and give word-identical answers on 213 of
-220 questions. An image request prefills in about 4 s and decodes at about
+220 questions. An image request prefills in about 4 s and, without the
+drafter, decodes at about
 23 tok/s.
 
 With our own ROCMFP MIX conversion of the same checkpoint (per-expert
@@ -260,7 +268,14 @@ importance matrix, the shipped recipe above), on a Strix Halo alone at top-k 6:
   and one-to-four-image sets are all correct.
 - With the published DSpark drafter and fused decode and verify, text decodes
   at 25 to 37 tok/s on 256-token answers (30 mean), as fast as the shipped
-  text model; image requests decode without the drafter at about 22 tok/s.
+  text model.
+- Image requests decode with the DSpark drafter too: on 12 images with
+  256-token answers, 13.3 s per answer (30 tok/s after the first token)
+  against 15.7 s (22 tok/s) without it. Capturing the drafter's features
+  during prefill adds about 0.7 s before the first token, so one-word answers
+  come back slightly later. The 220 questions score AI2D 86, ChartQA 55 and
+  40 with the drafter (209 answers identical to plain decode); one to four
+  images all correct.
 
 Not yet established:
 

server/src/common/vision/image_spans.h

@@ -33,6 +33,17 @@ inline const TokenSpan * image_block_at(ImageSpanView spans, uint64_t position)
     return nullptr;
 }
 
+// End of the last image block that overlaps [begin, end), or 0 when none does.
+inline uint64_t last_image_end_in(ImageSpanView spans, uint64_t begin, uint64_t end) {
+    uint64_t last = 0;
+    for (size_t i = 0; i < spans.size; ++i) {
+        const auto & span = spans.data[i];
+        if (span.block_begin >= end) break;
+        if (span.block_end > begin) last = span.block_end;
+    }
+    return last;
+}
+
 inline bool valid_image_spans(ImageSpanView spans, uint64_t prompt_size,
                               size_t max_images, uint64_t max_block_tokens) {
     if (spans.size > max_images || (spans.size && !spans.data)) return false;

server/src/deepseek4/deepseek4_backend.cpp

@@ -2641,8 +2641,9 @@ int DeepSeek4Backend::do_prefill(const std::vector<int32_t> & tokens,
                                   int snap_slot,
                                   int snap_pos,
                                   const DeepSeek4ImagePrompt * images) {
-    const bool capture_spec = !images && spec_enabled_ && spec_drafter_;
-    if (images) spec_feat_window_.clear();
+    // Image prompts capture DSpark features from their text chunks only: the
+    // image graph takes no capture hooks (see the chunking below).
+    const bool capture_spec = spec_enabled_ && spec_drafter_;
     const InferencePhase phase = deepseek4_roctx_prefill_phase(
         prefill_attention_mode_name(cfg_.prefill_mode));
     const DeepSeek4RoctxPhaseScope roctx_phase(phase);
@@ -2823,10 +2824,23 @@ int DeepSeek4Backend::do_prefill(const std::vector<int32_t> & tokens,
                 spec_snap_from, spec_snap_to);
         }
 
+        bool chunk_has_image = false;
         if (images) {
             n_tok = vision::atomic_image_chunk(images->spans(), uint64_t(pos), n_tok,
                                                uint64_t(n_total - i), image_capacity);
             if (!n_tok) return -1;
+            // An image batch cannot capture DSpark features, so end it at its
+            // last image: the text after the image then prefills (and
+            // captures) as ordinary chunks.
+            const uint64_t last_image_end =
+                vision::last_image_end_in(images->spans(), uint64_t(pos), uint64_t(pos + n_tok));
+            chunk_has_image = last_image_end != 0;
+            if (chunk_has_image && capture_spec && last_image_end < uint64_t(pos + n_tok)) {
+                n_tok = int(last_image_end - uint64_t(pos));
+            }
+            // The drafter reads the newest rows as one contiguous window, so
+            // rows from before an image cannot sit next to rows after it.
+            if (chunk_has_image) spec_feat_window_.clear();
         }
 
         // Bulk prompt graphs and the final DSpark feature-capture graph have
@@ -2873,7 +2887,7 @@ int DeepSeek4Backend::do_prefill(const std::vector<int32_t> & tokens,
         const bool capture_snapshot =
             !snapshot_saved && i < spec_snap_to &&
             i + n_tok > spec_snap_from;
-        if (capture_spec &&
+        if (capture_spec && !chunk_has_image &&
             (capture_final || capture_snapshot)) {
             spec_hooks.capture_layer_ids = &spec_drafter_->capture_layer_ids;
             spec_hooks.capture_out = &spec_cap;
@@ -3278,8 +3292,11 @@ GenerateResult DeepSeek4Backend::generate_from_state(
                 sampler_.rep_pen, sampler_.freq_pen, sampler_.pres_pen);
         }
     }
-    if (spec_enabled_ && spec_drafter_ && req.n_gen > 0 &&
-        !req.images && !req.force_ar_decode && !budget_requires_ar && !sampling_requires_ar) {
+    // An image prompt whose last image leaves no captured text rows gives the
+    // drafter no context to start from; decode that request plainly.
+    const bool image_without_draft_context = req.images && spec_feat_window_.empty();
+    if (spec_enabled_ && spec_drafter_ && req.n_gen > 0 && !image_without_draft_context &&
+        !req.force_ar_decode && !budget_requires_ar && !sampling_requires_ar) {
         if (last_logits_.empty()) {
             result.fail(GenerateErrorCode::DecodeFailed, "spec: no prefill logits");
             return result;

server/src/qwen35/qwen35_backend.cpp

@@ -1324,6 +1324,7 @@ DFlashTarget * Qwen35Backend::dflash_target() {
             qt->set_kvflash_pager(&kvflash_pager_);
         }
         qt->set_fast_rollback(cfg_.fast_rollback);
+        qt->set_rope_offset(&rope_delta_);
     }
     return dflash_target_.get();
 }
@@ -1508,9 +1509,9 @@ GenerateResult Qwen35Backend::generate_impl(const GenerateRequest & req,
                     req.n_gen, ar_n_gen, committed, cfg_.device.max_ctx);
             }
         }
-        // Speculative decoding takes rotary positions from KV positions, which
-        // an image prompt pulls apart, so image requests decode one by one.
-        if (cfg_.paged_attention || req.force_ar_decode || has_images) {
+        // Image requests speculate too: the verify target shifts its rotary
+        // positions by rope_delta_, and the drafter only proposes tokens.
+        if (cfg_.paged_attention || req.force_ar_decode) {
             decode_ok = do_ar_decode(committed, ar_n_gen, result.tokens, out_io,
                                      req.budget_hook,
                                      &result.budget_forced_close,

server/src/qwen35/qwen35_dflash_target.cpp

@@ -307,7 +307,7 @@ bool Qwen35DFlashTarget::verify_batch(
 
     // GGML M-RoPE positions are axis-major.
     std::vector<int32_t> pos(4 * n_tokens);
-    fill_qwen35_mrope_positions(pos.data(), base_pos, n_tokens);
+    fill_qwen35_mrope_positions(pos.data(), base_pos + rope_offset(), n_tokens);
     ggml_backend_tensor_set(sg_.positions, pos.data(), 0,
                             sizeof(int32_t) * pos.size());
 
@@ -452,7 +452,7 @@ bool Qwen35DFlashTarget::verify_tree(
     // M-RoPE axis-major positions: each node sits at committed + its depth.
     std::vector<int32_t> pos4(4 * N, 0);
     for (int i = 0; i < N_actual; i++) {
-        const int p = committed + (i == 0 ? 0 : tree.depths[i - 1]);
+        const int p = committed + rope_offset() + (i == 0 ? 0 : tree.depths[i - 1]);
         pos4[0 * N + i] = p;
         pos4[1 * N + i] = p;
         pos4[2 * N + i] = p;

server/src/qwen35/qwen35_dflash_target.h

@@ -93,6 +93,11 @@ class Qwen35DFlashTarget : public DFlashTarget {
     // so rollback_to() can restore recurrent state without replay.
     void set_fast_rollback(bool enabled) { fast_rollback_ = enabled; }
 
+    // Rotary positions run ahead of KV positions by this many after an image
+    // (image tokens take 2D positions). Points at the owner's per-request
+    // offset, which every prefill sets; null means zero.
+    void set_rope_offset(const int * offset) { rope_offset_ = offset; }
+
 private:
     TargetWeights & w_;
     TargetCache & cache_;
@@ -102,6 +107,8 @@ class Qwen35DFlashTarget : public DFlashTarget {
     int fa_window_;
     KvFlashPager * pager_ = nullptr;
     bool fast_rollback_ = false;
+    const int * rope_offset_ = nullptr;
+    int rope_offset() const { return rope_offset_ ? *rope_offset_ : 0; }
 
     // SpecLA (docs/SPECLA.md): true when the cache was
     // migrated with factor buffers. Capture-verify then runs the

server/src/server/http_server.cpp

@@ -4102,7 +4102,9 @@ void HttpServer::prepare_generation_inputs(
 
     inputs.request.prompt = prepared.tokens;
     inputs.request.images = prepared.images;
-    inputs.request.force_ar_decode = bool(prepared.images);
+    // Image requests may speculate; each backend decides (Qwen3.5 verifies at
+    // image-shifted rotary positions, DeepSeek4 drafts from the text after
+    // the last image).
     inputs.request.n_gen = inputs.generation_cap;
     inputs.request.sampler = req.sampler;
     inputs.request.do_sample = req.sampler.needs_logit_processing();

server/test/test_ds4v_image_integration.cpp

@@ -72,6 +72,11 @@ void validation_and_lookup() {
     std::array<TokenSpan, 5> too_many{};
     require(!valid_image_spans({too_many.data(), too_many.size()}, 100), "more than four images rejected");
     const std::vector<TokenSpan> spans{{10, 13, 18, 20}, {20, 20, 25, 25}, {30, 31, 33, 35}};
+    require(last_image_end_in(view(spans), 0, 10) == 0, "no image before the first block");
+    require(last_image_end_in(view(spans), 0, 11) == 20, "chunk reaching into the first image");
+    require(last_image_end_in(view(spans), 12, 26) == 25, "last overlapping image wins");
+    require(last_image_end_in(view(spans), 25, 30) == 0, "text between images");
+    require(last_image_end_in(view(spans), 34, 40) == 35, "chunk starting inside an image");
     require(valid_image_spans(view(spans), 35), "adjacent and separated blocks valid");
     require(!image_block_at(view(spans), 9), "text before block excluded");
     require(image_block_at(view(spans), 10) == &spans[0], "leading padding belongs to image block");