feat(phi4): run Phi-4-mini-instruct Q8_0 GGUF through ryzenai-corelib

[chizamd]

Runs Phi-4-mini-instruct from a Q8_0 GGUF, through AMD's
ryzenai-corelib. The weights are read straight from the GGUF — no ONNX
model, no tensor manifest, no converted weight file is produced or
shipped.

There is no separate tag for it. phi4-mini-it:4b resolves to the
artifacts the build's NPU generation can run: the NPU2/Q4NX entry on
stx, this GGUF entry on aie_next. flm list on a rai build shows only
what it can run.

112 files, +12,221 / −1,874 vs 0cfecb08. Packaging and CI are still
absent — see the known limitations at the end.

---

The file is memory-mapped and every tensor handed out as a span —
nothing copied, converted or cached to disk. RequireQ8/RequireF32
take a name and an expected shape, so all 161 tensors plus the rope
factors are resolved and shape-checked before the first device object
exists. Head counts, rope base and the short-factor table come from
the file's own metadata; the architecture is a fixed C++ constant block
that the GGUF metadata, config.json and tokenizer_config.json are
cross-checked against. A mismatched package fails at load with an exact
diagnostic, never mid-generation.

tokenizer_config.json is parsed once by the frontend and passed down
through BackendContext, so the model directory layout stays the
frontend's knowledge and a backend never opens the directory itself.

Requantizing the 161 weights from Q8_0 to group-64 is effectively all of
model load. Two things address it.

The creates are independent — each reads its own mapped range and
produces its own object — so they run across a pool of 8. The per-create
thread hint stays at corelib's default of one so the two forms of
parallelism don't multiply. The pool is threads FastFlowLM owns, so it
is unaffected by thread limits the surrounding environment may impose on
the packer.

On top of that, the packed bytes are cached. All 161 blobs go in one
file with a JSON index written last and renamed into place, so a data
file without a matching index is never used. Both directions walk the
same slot order, so an index entry always refers to the weight it was
written from — the creates finish out of order but are assigned by slot,
not by completion. The key is the GGUF's size and write time, the
corelib version, the group size and the weight count; not the GGUF's
content, because hashing 4 GB costs more than the packing the cache
exists to avoid. That is safe here because corelib independently rejects
a slice whose length isn't exactly what the descriptor packs to, so a
stale-but-plausible cache cannot quietly become the wrong weights. A
miss, a stale key, an unreadable index or a failed bind all fall back to
packing, and a partial bind releases everything rather than leaving half
a model. A cache that no longer matches is deleted when that is known,
not whenever the next write happens to succeed. FLM_RAI_WEIGHT_CACHE
redirects it, or disables it with 0.

Every device tensor — hidden/residual/skip, Q, K, attention, the 32 K
and V caches, rope tables — is created once at load, sized to the
maximum padded extents the shape plan interrogates out of corelib's own
padding helpers. The plan is queried only at corelib's nine real
execution buckets (1, 64, 128, 256, 512, 1024, 2048, 3072, 4096) and
every logical row maps to its next bucket, so nothing re-enters the
helpers on the hot path and nothing allocates per token.

Per layer the device does everything except the embedding gather: Q/K
matmul, V matmul written directly into a tensor window of the V cache
at the current position (no host-side KV scatter), flat_mha against
the caches with the rope tables, O projection, fused SSMLP with a
residual/skip ping-pong. Two synchronizes per token — one after all
32 layers, one after the head — not one per layer.

The NPU generation is not detected. A build carries engines for exactly
one of them, FLM_ENABLE_RAI picks which at compile time, and
utils::build_npu_platform() is a constexpr that reports the choice.
The catalog in model_list.json declares which generations each model
supports and model_list prunes itself to the one the build was made
for, so flm list works on a machine with no NPU without asking the
hardware anything.

Backend selection is separate and explicit: a build links one kernel
flow, and that is the default, never a filename or a quantization level.
The catalog does not name a backend — details.execution_backend is
retired, and a test asserts it stays that way. Once this entry is selected there is
no fallback — a missing, unloadable or wrong-version corelib fails
the load with a diagnostic rather than quietly running on CPU or NPU2.
A rai build links corelib in, because the NPU device the process
shares comes from corelib's GetDevice(); an exact 0.5.0 match is
required while corelib is pre-1.0, enforced at compile time and again at
run time. Both checks read the vendor header's own
RYZENAI_CORELIB_VERSION_* macros rather than repeating the triple, so
the compile-time #error, the run-time comparison, the fake, the tests
and the weight-cache key cannot drift apart on the next bump.

Once work has reached the device, any throw drains with a synchronize
and marks the engine poisoned; every later entry point refuses
rather than continuing on indeterminate state.

File	Source	Revision
Phi-4-mini-instruct.Q8_0.gguf	unsloth/Phi-4-mini-instruct-GGUF
78eb92a4
tokenizer.json, tokenizer_config.json, config.json
microsoft/Phi-4-mini-instruct	cfbefacb

All four are SHA-256 verified before the download is promoted.
Verification is a pull-time concern: flm pull and flm check verify
in full, while run and serve check presence and version rather than
re-hashing 4 GB on every launch.

The usable window is 4095 tokens (prompt + output), so the largest
admissible prompt is 4094; over-capacity requests are rejected with HTTP
400 before submission. The 4096 ceiling is a correctness boundary, not a
buffer size: it is Phi-4-mini's rope.scaling.original_context_length,
LongRoPE selects factors by sequence length, and only the short branch
is derived here — enforced at load, which fails unless the GGUF reports
exactly 4096. The further step down to 4095 is this frontend's own
conservatism so an admitted request can always finish; it is not imposed
by corelib.

Model loaded in N s on both the CLI and server load paths.
FLM_RAI_PROFILE_LOAD=1 breaks the rai load into shape planning, GGUF
resolution, host preparation, weight requantization and device
allocation, and says whether the weights came from the cache, what
writing one cost, and how much stale cache was reclaimed. Decode timing
is recorded on this backend, so /status, /verbose and the
Ollama-compatible eval_duration report real figures.

corelib 0.5.0 moved three things that reach into this engine. The row
count left every dispatch: M now comes from the operand's own shape, and
each operand is checked against its OWN padded extent rather than through
the allocation it sits in -- so a forward opens one view per operand per
op, at the extent the plan interrogated for it, over device tensors that
are still created once at load. The padding helpers gained the stream,
because the PDI pair a stream was opened with selects the kernel set and
a shape can exist under one pair and not the other; the shape plan is
therefore built after the stream rather than in the constructor's
initializer list. And the PDI pair itself is now required and has no
default: Phi-4 is one of the older families, so p1 prefill / p16 token,
pinned in phi4_rai_constants.hpp with the rest of the model's facts.
has_device_context() became get_device(), NULL for no NPU.

---

Three standalone suites, all hardware-free except where noted:

• phi4_rai — 9 targets on a fake corelib implementing
  the 0.5.0 C ABI in-process plus a deterministic GGUF v3 fixture
  builder: ABI/version/path/lifetime, parsing and corruption, host ops,
  shape plan, engine load and dispatch sequencing with object-leak and
  concurrency checks, weight-cache round trip / staleness / reclaim,
  frontend routing and request lifecycle, a compile-gate proving the OFF
  build carries no corelib, and the downloader including legacy-entry
  compatibility. 9/9 on an aie_next host, with test_real_corelib
  executing against the real corelib DLL rather than skipping; 8/9
  elsewhere, where that one test skips for want of hardware.
• model_backend — the backend registry and resolution rules, on
  stubs. 9/9.
• model_list_platform — catalog platform filtering and override
  merging. 6/6.

run_real_rai_acceptance.ps1 drives the hardware matrix end to end and
writes a machine-readable record. Its provenance step degrades to
unknown when a revision cannot be read rather than failing the run:
it used to call .Trim() on the output of a git rev-parse that had
printed nothing, which took the whole matrix down after it had already
passed.

---

Two aie_next machines, measuring different things.

Machine A — full acceptance matrix, passed: true, 0 failures:
pinned pull/check; 10/10 fresh-process CLI load-and-generate cycles;
/api/chat and /v1/chat/completions both 200, streaming and
non-streaming; cancellation clean with the next request served; 4095
admitted and 4096 rejected with HTTP 400 before submission; no CPU or
NPU2 fallback in any log. Generation figures: cold TTFT 4.21 s, warm
TTFT 65.0 ms, decode 21.3 tok/s over REST and 35.8 tok/s in a warm CLI
session.

Machine B (a 3-column aie_next part) — model load, with the current
packer:

Weight phase	Range
Packing (no cache)	3.50 – 13.51 s, median 8.76
Cache hit	0.07 – 3.57 s, median 3.56

First load additionally writes a 2.0 GB cache in ~1–2 s. The 50× spread
within cache hits is page-cache warmth. Load on this machine varies
about 4× run to run on an otherwise idle box, so these are ranges rather
than single figures — a single sample of this phase is not meaningful.

docs/docs/benchmarks/phi4_results.md keeps the two machines separate
and says plainly which figures were measured where, and which were not
re-measured after the packer changed.

---

1. Greedy decoding is not reproducible across loads. Eight loads of
   the same model answering the same prompt at temperature=0, top_k=1
   produced 2 distinct answers; serialising the packer produced 3, so this
   is not caused by the concurrent creates. It is consistent with the
   requantization refit not being bit-reproducible, and is expected to
   disappear when dedicated ELFs remove the refit.
2. The acceptance matrix was re-run in full after the move to corelib
   0.5.0, on an aie_next host: passed: true, 0 failures, 10/10
   fresh-process CLI load-and-generate cycles, REST both streaming and
   non-streaming, cancellation clean, the capacity boundary rejected
   before submission. Cold TTFT 4.65 s, warm TTFT 35.1 ms, decode 34.6
   tok/s. The three unit suites are 11/11 on the same box, with
   test_real_corelib running against the real DLL rather than skipping.
3. calculate_file_sha256 is ~8× slower than the work requires —
   ~28 s over 4 GB where Get-FileHash takes 3.67 s, because it uses a
   portable pure-C++ SHA-256. No longer on the startup path, but every flm pull and flm check pays it, for every model. Shared pull code,
   unrelated to this backend, left for its own change.
4. A shared FLM_RAI_WEIGHT_CACHE directory holds one cache, so
   alternating two models there repacks each time. The default — beside the
   model — does not have this.
5. No packaging. MSI, WiX and Inno are untouched; this is a
   rai-enabled developer build.
6. No CI builds or runs any of this, and the release preset does not
   compile the rai path.
7. Windows only, Q8_0 only — no Q4_0/Q4_K/Q6_K, no mixed
   quantization, no generic GGUF runtime, no other model family.
8. Usable context is 4095, against 128k for the model and 32k for
   the stx entry.

#706 reaches the same backend through an **ONNX manifest

• packed weights** pipeline and carries packaging this branch does not.
  This is the GGUF-direct approach: smaller surface, no manifest
  generator, no converted artifacts, no Python at runtime. They are
  alternatives; only one should land.

---

Co-authored-by: Claude Opus 5 (1M context) noreply@anthropic.com
Co-authored-by: alfxu_amdeng Alfred.Xu@amd.com

---

🤖 Generated with Claude Code