Qwen3.8 27b

[michyu-amd]

Summary

Adds qwen3-8mtp — the Qwen3.8-27B engine — as a new model family, and adds the
runtime plumbing for speculative decoding driven by the model's MTP draft head.

Qwen3.8-27B is a hybrid stack: 64 decoder layers in the pattern
[linear, linear, linear, full] x 16 (48 GatedDeltaNet + 16 full attention),
plus a 1-layer MTP head used as a draft model. The engine dispatches to the
NPU2 bitstreams checked in under src/xclbins/Qwen3.8-27B-NPU2/, with CPU
reference kernels as the fallback path.

Model is registered at qwen3-8mtp:27b → FastFlowLM/Qwen3.8-27B-NPU2
(Q4_K, 17.1 GB footprint, 4096 default context, 256 max prefill).

What's in it

New engine

• src/include/models/qwen3_8mtp/qwen3_8mtp_npu.hpp + src/lib/xrt/libqwen3_8mtp_npu.so
• src/common/AutoModel/modeling_qwen3_8mtp.cpp — AutoModel wrapper
• src/common/AutoModel/modeling_qwen3_8mtp_image.cpp — vision tower host-side
  pipeline (active only when the checkpoint ships vision_weight.q4nx; images
  are rejected loudly on a text-only checkpoint rather than silently dropped)
• src/test/qwen3_8mtp_npu/ — standalone driver for scoring the engine against
  a reference dump
• Registration in all_models.hpp, automodel.hpp, CMakeLists.txt, model_list.json

Speculative decoding

• Three new virtuals on causal_lm, all defaulted (not pure) so the other 16
  engines are unchanged in behaviour: supports_speculation(),
  speculate(last_token, max_draft), last_speculation_prime_us().
• AutoModel::_shared_generate() takes the speculative path only when the
  engine supports it and the sampler is truly greedy — top_k == 1 and all
  three penalties neutral. Acceptance is an exact argmax compare, so running it
  under any other sampler would silently substitute greedy decoding for the
  user's configured sampler. The output would still be fluent, so nothing
  downstream would ever flag it; all four conditions are load-bearing.
• Per-token streaming, token history, EOS and length-limit handling are factored
  into a single consume() lambda shared by both paths, so the speculative
  branch cannot drift into emitting text past a stop token.
• The first speculative cycle after each prefill primes the draft head's KV
  cache over the prompt window. That is prefill-shaped work happening inside a
  speculate() call with the decode clock running, and it scales with prompt
  length — leaving it there makes decode tok/s a function of prompt size. The
  engine reports the split and the runtime moves it from DECODING_TIME to
  PREFILL_TIME.

Profiling

• show_profile() re-enables the sampling / token-encode / token-decode rows
  (previously commented out) and adds an Untimed remainder row.
  This was added because the gap was real and invisible: Qwen3.8's think
  preamble spent ~13 s of a 24 s run in four forward() calls that
  _shared_generate()'s DECODING_TIME.reset() then discarded.
  Caveat documented in-place: the narrow rows only compare against Total on a
  single-turn run, since DECODING_TIME resets per turn while PREFILL_TIME
  and TOTAL_TIME accumulate. A multi-turn session over-reports Untimed, and
  the row prints signed rather than clamping, so that case announces itself.
• profiler::add_time(int64_t) for transferring an already-measured interval
  between buckets (clamped at zero).

minja

• is undefined test. Upstream minja (3e4c61c) implements is defined but not
  its negation, and throws on the test name before looking at the value, so
  x is undefined fails even when x is set. Qwen3.8-27B's chat template opens
  with {%- if enable_thinking is undefined or enable_thinking is true %}.

Blocking before merge: stale engine vtables

Adding virtuals to causal_lm is a lockstep change. Every engine .so
emits its own vtable and flm loads those prebuilt; appending to the base class
renumbers the vtable, so any .so not rebuilt keeps one that is short by the
added slots, and the first virtual call past the end segfaults — after a clean
prefill, with no diagnostic. A link check cannot catch this: the vtable symbol
still resolves, it is merely the wrong size.

This branch rebuilt only libqwen3_8mtp_npu.so. The other 16 are stale:

$ for f in src/lib/xrt/lib*_npu.so; do readelf -sW "$f" | grep -E '_ZTV.*_npu'; done
libqwen3_8mtp_npu.so   144  _ZTV14qwen3_8mtp_npu   <-- rebuilt
libqwen3_npu.so        120  _ZTV9qwen3_npu         <-- stale
libqwen3_6_moe_npu.so  120  _ZTV15qwen3_6_moe_npu  <-- stale
libgemma4_12b_npu.so   120  _ZTV14gemma4_12b_npu   <-- stale
... (13 more, all 120)

All engine .so files under src/lib/xrt/ need to be regenerated and re-copied
in this PR before it is safe to merge — otherwise every model except
Qwen3.8-27B crashes on its first decode step. Verify with the readelf command
above rather than with "it compiled" or "it linked".

(Note: the ABI comment in causal_lm.hpp says the change takes the vtable from
120 to 136 bytes — it is 144, since three virtuals were added, not two. Worth
correcting so the number in the comment matches what readelf prints.)

Also needs a look before merge

• prompt.txt at the repo root looks like a stray scratch file (an LLM
  prompt about dispatch-count profiling) rather than something intended to ship.
  Probably wants deleting.
• .claude/ in .gitignore — fine if the project wants it, flagging only
  because it is unrelated to the model.

Testing

• src/test/qwen3_8mtp_npu/ driver runs a text turn and (with -i) one
  multimodal turn, scoring intermediates against a reference dump.
• Speculation statistics are exposed engine-side (mean_accepted_length(),
  draft_hit_rate(), speculation_cycles(), speculation_timing(),
  speculation_phase_fraction()) because a broken draft path is otherwise
  silent — verify overrides every rejected draft with the base model's own
  argmax, so bad drafting still produces correct text and merely burns time.
  speculation_phase_fraction() is deliberately normalised by measured cycle
  time, not by the sum of the parts, so a double-counted phase pushes it above
  1.0 instead of hiding itself.

Still to confirm on hardware:

• All 16 other engines rebuilt; readelf vtable sizes agree
• Non-speculative models unchanged (regression run on qwen3, gemma4)
• Draft hit rate / accepted-length on a representative prompt set
• Untimed row sanity-checked on a single-turn run