Skip to content

feat(qwen35): serve image requests in the concurrent batch - #759

Merged
davide221 merged 2 commits into
mainfrom
feat/qwen35-batched-images
Sep 24, 2026
Merged

feat(qwen35): serve image requests in the concurrent batch#759
davide221 merged 2 commits into
mainfrom
feat/qwen35-batched-images

Conversation

@davide221

@davide221 davide221 commented Sep 23, 2026 •
edited by cubic-dev-ai Bot
Loading

Copy link
Copy Markdown
Contributor

What

Qwen3.5 / Qwen3.8 servers can now take image requests with concurrent scheduling (--paged-attention --max-concurrency N). Until now a vision projector turned batching off for everyone. Image requests now prefill and decode in the shared batch like text, with the drafter.

Changes

  • Engine interface. SeqEngine gains supports_images() and admit_images(). By default an engine refuses image requests.
  • Scheduler. Image requests go to admit_images(). They never get a prefix-cache plan, because tokens alone do not identify an image.
  • Qwen35SeqEngine:
    • Encodes the images at admission, before claiming a slot, so a failed encode leaves nothing to unwind.
    • Keeps the payload, rows and rotary offset with the slot until it retires.
    • Every prefill chunk that covers an image gets the image rows and M-RoPE positions again, so a re-prefill after eviction still sees the images.
    • Shifts decode and chain-verify rotary positions by the slot's offset.
  • Server and feature gate. Image input is enabled when the engine supports it. The gate allows --mmproj with --max-concurrency for Qwen3.5; DeepSeek4 still needs one request at a time.
  • Docs. docs/image-input.md updated.

Results

lucebox6, one R9700, Qwen3.8-27B-IQ4_XS-pure + Q8_0 projector + DFlash2 drafter:

check single-request server --paged-attention --max-concurrency 4
sanity set 6/6 6/6
one to four images per request 9/9 9/9
1 image request, 256 tokens 78 tok/s 81 tok/s
2 at once 73 tok/s total 104 tok/s
4 at once 77 tok/s, 13.3 s 149 tok/s, 6.9 s

Each concurrent answer describes its own chart; nothing leaks between slots.

Not covered

  • No unit test. A scheduler test would need an image-capable fake backend in the server harness; validation here is on the box.
  • Encoding blocks the batch briefly. It runs at admission on the scheduler thread, so live slots pause for the encode, about 0.3 to 0.6 s per image.

🤖 Generated with Claude Code

Review in cubic

mrciffa @claude
feat(qwen35): serve image requests in the concurrent batch
bfe6f4d 
A vision projector used to switch concurrent sequence scheduling off for
image input. The batched engine now accepts image requests:

- SeqEngine gains supports_images() and admit_images(); the default
  refuses. The scheduler routes image requests there and never gives
  them a prefix-cache plan (tokens alone do not identify an image).
- Qwen35SeqEngine encodes the images at admission, keeps the payload,
  rows and rope offset with the slot until it retires, overwrites image
  rows and writes the image's M-RoPE positions in every prefill chunk
  that covers an image (so an eviction re-prefill sees them again), and
  shifts decode and chain-verify rotary positions by the slot's offset.
- The server enables image input when the engine supports it; the
  feature gate allows --mmproj with --max-concurrency for Qwen3.5.
  DeepSeek4 still requires one request at a time.

lucebox6 R9700, Qwen3.8-27B-IQ4_XS-pure + Q8_0 projector + DFlash2,
--paged-attention --max-concurrency 4: sanity 6/6, one to four images
9/9 (same as single-request); 1/2/4 concurrent 256-token image answers
81/104/149 tok/s total (single-request server 78/73/77), 4 answers in
6.9 s instead of 13.3 s, each answer about its own chart.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@davide221 davide221 mentioned this pull request Sep 23, 2026
Merged
mrciffa @claude
fix(qwen35): claim the slot before encoding a batched image request
87bce9e 
A busy pool defers the request and retries it; encoding first reran the
vision tower on every retry. A failed encode now retires the slot.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@davide221
davide221 merged commit 81a9966 into main Sep 24, 2026
8 checks passed
davide221 added a commit that referenced this pull request Sep 24, 2026
@davide221 mrciffa @claude
feat(deepseek4): serve image requests in the concurrent batch (#760)
3d8da89 
* feat(qwen35): speculative decoding for image requests

Image tokens take 2D rotary positions, so after an image the rotary
position runs rope_delta_ ahead of the KV position. AR decode already
applied that offset; the DFlash verify target did not, so the HTTP layer
and the backend forced every image request to plain AR decode.

Qwen35DFlashTarget now reads the backend's per-request rope_delta_ and
shifts the M-RoPE positions of chain and tree verify by it (zero for text,
so text requests are unchanged). The blanket AR force for images is
dropped from the HTTP layer and the Qwen3.5 backend; DeepSeek4 keeps its
own image guard and still decodes image requests AR.

R9700, Qwen3.8-27B-IQ4_XS-pure + Q8_0 projector + DFlash2, 12 images,
256-token answers: 4.03 s per answer (76 tok/s) vs 7.58 s (36 tok/s)
before; llama.cpp with the same drafter 5.54 s, without 8.36 s. Image
eval 188/220 unchanged (218 answers identical), 1-4 images 9/9.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(deepseek4): DSpark speculative decoding for image requests

The image prefill graph takes no DSpark capture hooks, so image requests
skipped feature capture and decoded AR. Image chunks now end at their last
image, the text after the image prefills and captures as ordinary chunks,
and the feature window is cleared at each image chunk so the drafter
always reads one contiguous tail. With that, image requests take the
DSpark path like text.

Strix Halo alone, Vision-Exp ROCMFPX MIX, published DSpark launch, 12
images, 256-token answers: 13.3 s per answer (30.4 tok/s) vs 15.7 s
(22.1 tok/s); first token about 0.7 s later from the capture band. 220
image questions 181 (AI2D 86, ChartQA 55/40), 209 identical to plain
decode; one to four images 9/9; text decode unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(deepseek4): run the vision encoder on a second GPU and stream images into prefill

--mmproj-device hip:N loads the DS4V encoder on another GPU in the one-GPU
layout (the R9700 next to a Strix Halo holding the model). Its scratch is
charged to that GPU. The encoder then runs image by image on a background
thread and publishes each image's rows; prefill waits per chunk only for
the images that chunk contains, so the Strix Halo prefills image k while
the R9700 encodes image k+1. Failure or cancellation releases waiters and
the thread is joined before the request, shutdown or park returns.

Requests may carry up to 16 images (one shared constant for the HTTP
transport, DS4V and Qwen3.5), and each request logs its encode time.

lucebox6, Strix Halo decoder, published DSpark launch, ChartQA charts,
time to first token, Strix encoder -> R9700 encoder streamed:
1 image 2.97 -> 2.94 s, 4 images 11.2 -> 9.1 s, 8 images 27.7 -> 19.2 s,
16 images (4,358 tokens) 51.8 -> 34.6 s. Answers identical to the
sequential encode.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(qwen35): serve image requests in the concurrent batch

A vision projector used to switch concurrent sequence scheduling off for
image input. The batched engine now accepts image requests:

- SeqEngine gains supports_images() and admit_images(); the default
  refuses. The scheduler routes image requests there and never gives
  them a prefix-cache plan (tokens alone do not identify an image).
- Qwen35SeqEngine encodes the images at admission, keeps the payload,
  rows and rope offset with the slot until it retires, overwrites image
  rows and writes the image's M-RoPE positions in every prefill chunk
  that covers an image (so an eviction re-prefill sees them again), and
  shifts decode and chain-verify rotary positions by the slot's offset.
- The server enables image input when the engine supports it; the
  feature gate allows --mmproj with --max-concurrency for Qwen3.5.
  DeepSeek4 still requires one request at a time.

lucebox6 R9700, Qwen3.8-27B-IQ4_XS-pure + Q8_0 projector + DFlash2,
--paged-attention --max-concurrency 4: sanity 6/6, one to four images
9/9 (same as single-request); 1/2/4 concurrent 256-token image answers
81/104/149 tok/s total (single-request server 78/73/77), 4 answers in
6.9 s instead of 13.3 s, each answer about its own chart.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(deepseek4): serve image requests in the concurrent batch

DS4V image blocks need whole-block bidirectional prefill; the batched
engine's gathered step is 16 causal rows. So an image request admitted to
the batch is prefilled up to its last token on the single-request sparse
path into a staging cache (the encoder streams from --mmproj-device when
set), that state is copied into the request's paged slot, and the last
token prefills in the batch, producing the first token through the normal
step. Decode then runs alongside every other sequence.

- import_deepseek4_paged_slot copies the 128-row raw ring, the completed
  compressed and indexer rows through the slot's block table, and the
  compressor states, checking layouts and row counts.
- DeepSeek4SeqEngine::admit_images seeds the slot with
  seed_restored_prefix and retires it on any failure.
- do_prefill can stop after a prefix and takes its attention mode from the
  cache it fills (identical to the config for the single-request cache).
- Paged serving with --mmproj creates the staging cache (sparse) and lets
  one image request per slot through the image gate; the feature gate
  allows DeepSeek4 --mmproj with --paged-attention batching.

lucebox6, Strix Halo decoder, R9700 encoder, 4 slots: sanity 2/2,
one to four images 9/9; 4 concurrent 256-token image answers in 39.1 s
(26.2 tok/s total; 1/2 at once: 17.4/21.7 tok/s), 2 images + 2 texts in
33.1 s (30.9 tok/s), 4 texts 37.9 tok/s.

Stacked on #758, #754 and #759.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* perf(deepseek4): prefill concurrent image requests in one shared pass

Image admissions now only encode their images and seed their slot; the
next batched step prefills every pending image request together.
deepseek4_prefill_multi runs one layer-major pass over several
sequences: attention per sequence against its own staging cache (with
its image masks), HC mixing and the MoE FFN once over all rows, so each
layer's expert weights are read once for every request in the pass. The
states are then copied into the paged slots as before. A failed request
fails only its own slot.

lucebox6, Strix Halo + R9700 encoder, 4 slots: 4 image requests share
one 1,020-row pass (7.1 s); 4 concurrent 256-token image answers
39.1 -> 35.2 s (26.2 -> 29.1 tok/s), 2+2 mixed 30.9 -> 31.5 tok/s,
sanity 2/2, one to four images 9/9.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(vision): address review on image speculative decode

- DeepSeek4 falls back to plain decode when an image request has no text
  after the last image to seed the drafter window.
- vision::last_image_end_in replaces the inline span loop in do_prefill,
  with unit checks.
- Clarify the http_server comment and the image-input doc figures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(qwen35): claim the slot before encoding a batched image request

A busy pool defers the request and retries it; encoding first reran the
vision tower on every retry. A failed encode now retires the slot.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(deepseek4): keep exceptions inside the image stream thread

A throw in the --mmproj-device encoder thread would terminate the server;
it now fails the stream so prefill stops waiting. Encode-time logs print
only on success, and <thread> joins the system includes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(deepseek4): claim the slot before encoding a batched image request

A busy pool defers the request and retries it; encoding first reran the
encoder on the scheduler thread on every retry. Names the layer-major
prefill minimum (DS4_MIN_LAYER_MAJOR_PREFILL_TOKENS) instead of a bare 5.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(vision): answer 503 when the image request gate is full

With batched DS4V serving, image requests beyond one per slot were refused
with HTTP 400, which clients do not retry. prepare_images now returns an
ImagePrepareStatus (ok, invalid, busy) and the server maps busy to 503.
Adds a gate capacity test and brings the DS4V batching docs up to the
shared staged prefill (4 image answers 35 s, 29 tok/s).

lucebox6: 6 concurrent image requests on 4 slots -> 4 answered in 35.1 s,
2 x 503; test_server_unit 595/595, DS4V image unit tests pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(deepseek4): queue image requests instead of refusing them

The image request gate refused every image request beyond one per slot
(one per backend without batching). A waiting request holds only its
preprocessed patches, a few MB per image, and its encoded rows exist only
once it runs, so the slots already bound them: DeepSeek image requests now
wait in the scheduler queue like text and Qwen image requests. The gate
and its lease are removed; short host memory answers 503 (busy).

lucebox6: batched, 6 image requests on 4 slots all answered (4 at 35.1 s,
2 queued at 59.6 s), one encode each; single-request server, 2 at once
both answered (14.5 s, 30.4 s); sanity 2/2; test_server_unit 595/595.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: mrciffa <davide@lucebox.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@davide221