From 1313ef8e81820f415e9ae73351aed2794d4b11c3 Mon Sep 17 00:00:00 2001 From: Chi Zhang Date: Thu, 17 Sep 2026 15:05:09 -0700 Subject: [PATCH 01/17] feat(phi4): run Phi-4-mini-instruct Q8_0 GGUF through ryzenai-corelib MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Runs **Phi-4-mini-instruct from a Q8_0 GGUF**, through AMD's `ryzenai-corelib`. The weights are read straight from the GGUF — no ONNX model, no tensor manifest, no converted weight file is produced or shipped. There is **no separate tag for it**. `phi4-mini-it:4b` resolves to the artifacts the build's NPU generation can run: the NPU2/Q4NX entry on `stx`, this GGUF entry on `aie_next`. `flm list` on a rai build shows only what it can run. 112 files, +12,221 / −1,874 vs `0cfecb08`. Packaging and CI are still absent — see the known limitations at the end. --- The file is memory-mapped and every tensor handed out as a `span` — nothing copied, converted or cached to disk. `RequireQ8`/`RequireF32` take a name *and* an expected shape, so all 161 tensors plus the rope factors are resolved and shape-checked **before the first device object exists**. Head counts, rope base and the short-factor table come from the file's own metadata; the architecture is a fixed C++ constant block that the GGUF metadata, `config.json` and `tokenizer_config.json` are cross-checked against. A mismatched package fails at load with an exact diagnostic, never mid-generation. `tokenizer_config.json` is parsed once by the frontend and passed down through `BackendContext`, so the model directory layout stays the frontend's knowledge and a backend never opens the directory itself. Requantizing the 161 weights from Q8_0 to group-64 is effectively all of model load. Two things address it. The creates are independent — each reads its own mapped range and produces its own object — so they run across a pool of 8. The per-create thread hint stays at corelib's default of one so the two forms of parallelism don't multiply. The pool is threads FastFlowLM owns, so it is unaffected by thread limits the surrounding environment may impose on the packer. On top of that, the packed bytes are cached. All 161 blobs go in one file with a JSON index written last and renamed into place, so a data file without a matching index is never used. Both directions walk the same slot order, so an index entry always refers to the weight it was written from — the creates finish out of order but are assigned by slot, not by completion. The key is the GGUF's size and write time, the corelib version, the group size and the weight count; **not** the GGUF's content, because hashing 4 GB costs more than the packing the cache exists to avoid. That is safe here because corelib independently rejects a slice whose length isn't exactly what the descriptor packs to, so a stale-but-plausible cache cannot quietly become the wrong weights. A miss, a stale key, an unreadable index or a failed bind all fall back to packing, and a partial bind releases everything rather than leaving half a model. A cache that no longer matches is deleted when that is known, not whenever the next write happens to succeed. `FLM_RAI_WEIGHT_CACHE` redirects it, or disables it with `0`. Every device tensor — hidden/residual/skip, Q, K, attention, the 32 K and V caches, rope tables — is created once at load, sized to the maximum padded extents the shape plan interrogates out of corelib's own padding helpers. The plan is queried only at corelib's nine real execution buckets (1, 64, 128, 256, 512, 1024, 2048, 3072, 4096) and every logical row maps to its next bucket, so nothing re-enters the helpers on the hot path and nothing allocates per token. Per layer the device does everything except the embedding gather: Q/K matmul, V matmul written **directly into a tensor window of the V cache at the current position** (no host-side KV scatter), `flat_mha` against the caches with the rope tables, O projection, fused SSMLP with a residual/skip ping-pong. **Two synchronizes per token** — one after all 32 layers, one after the head — not one per layer. The NPU generation is not detected. A build carries engines for exactly one of them, `FLM_ENABLE_RAI` picks which at compile time, and `utils::build_npu_platform()` is a `constexpr` that reports the choice. The catalog in `model_list.json` declares which generations each model supports and `model_list` prunes itself to the one the build was made for, so `flm list` works on a machine with no NPU without asking the hardware anything. Backend selection is separate and explicit: a build links one kernel flow, and that is the default, never a filename or a quantization level. The catalog does not name a backend — `details.execution_backend` is retired, and a test asserts it stays that way. Once this entry is selected there is **no fallback** — a missing, unloadable or wrong-version corelib fails the load with a diagnostic rather than quietly running on CPU or NPU2. A rai build links corelib in, because the NPU device the process shares comes from corelib's `GetDevice()`; an exact `0.5.0` match is required while corelib is pre-1.0, enforced at compile time and again at run time. Both checks read the vendor header's own `RYZENAI_CORELIB_VERSION_*` macros rather than repeating the triple, so the compile-time `#error`, the run-time comparison, the fake, the tests and the weight-cache key cannot drift apart on the next bump. Once work has reached the device, any throw drains with a synchronize and marks the engine **poisoned**; every later entry point refuses rather than continuing on indeterminate state. | File | Source | Revision | |---|---|---| | `Phi-4-mini-instruct.Q8_0.gguf` | `unsloth/Phi-4-mini-instruct-GGUF` | `78eb92a4` | | `tokenizer.json`, `tokenizer_config.json`, `config.json` | `microsoft/Phi-4-mini-instruct` | `cfbefacb` | All four are SHA-256 verified before the download is promoted. Verification is a pull-time concern: `flm pull` and `flm check` verify in full, while run and serve check presence and version rather than re-hashing 4 GB on every launch. The usable window is **4095** tokens (prompt + output), so the largest admissible prompt is 4094; over-capacity requests are rejected with HTTP 400 before submission. The 4096 ceiling is a correctness boundary, not a buffer size: it is Phi-4-mini's `rope.scaling.original_context_length`, LongRoPE selects factors by sequence length, and only the short branch is derived here — enforced at load, which fails unless the GGUF reports exactly 4096. The further step down to 4095 is this frontend's own conservatism so an admitted request can always finish; it is not imposed by corelib. `Model loaded in N s` on both the CLI and server load paths. `FLM_RAI_PROFILE_LOAD=1` breaks the rai load into shape planning, GGUF resolution, host preparation, weight requantization and device allocation, and says whether the weights came from the cache, what writing one cost, and how much stale cache was reclaimed. Decode timing is recorded on this backend, so `/status`, `/verbose` and the Ollama-compatible `eval_duration` report real figures. corelib 0.5.0 moved three things that reach into this engine. The row count left every dispatch: M now comes from the operand's own shape, and each operand is checked against its OWN padded extent rather than through the allocation it sits in -- so a forward opens one view per operand per op, at the extent the plan interrogated for it, over device tensors that are still created once at load. The padding helpers gained the stream, because the PDI pair a stream was opened with selects the kernel set and a shape can exist under one pair and not the other; the shape plan is therefore built after the stream rather than in the constructor's initializer list. And the PDI pair itself is now required and has no default: Phi-4 is one of the older families, so p1 prefill / p16 token, pinned in phi4_rai_constants.hpp with the rest of the model's facts. `has_device_context()` became `get_device()`, NULL for no NPU. --- Three standalone suites, all hardware-free except where noted: - **`phi4_rai`** — 9 targets on a **fake corelib** implementing the 0.5.0 C ABI in-process plus a deterministic **GGUF v3 fixture builder**: ABI/version/path/lifetime, parsing and corruption, host ops, shape plan, engine load and dispatch sequencing with object-leak and concurrency checks, weight-cache round trip / staleness / reclaim, frontend routing and request lifecycle, a compile-gate proving the OFF build carries no corelib, and the downloader including legacy-entry compatibility. **9/9 on an aie_next host**, with `test_real_corelib` executing against the real corelib DLL rather than skipping; 8/9 elsewhere, where that one test skips for want of hardware. - **`model_backend`** — the backend registry and resolution rules, on stubs. 9/9. - **`model_list_platform`** — catalog platform filtering and override merging. 6/6. `run_real_rai_acceptance.ps1` drives the hardware matrix end to end and writes a machine-readable record. Its provenance step degrades to `unknown` when a revision cannot be read rather than failing the run: it used to call `.Trim()` on the output of a `git rev-parse` that had printed nothing, which took the whole matrix down after it had already passed. --- Two aie_next machines, measuring different things. **Machine A** — full acceptance matrix, `passed: true`, 0 failures: pinned `pull`/`check`; 10/10 fresh-process CLI load-and-generate cycles; `/api/chat` and `/v1/chat/completions` both 200, streaming and non-streaming; cancellation clean with the next request served; 4095 admitted and 4096 rejected with HTTP 400 before submission; no CPU or NPU2 fallback in any log. Generation figures: cold TTFT 4.21 s, warm TTFT 65.0 ms, decode 21.3 tok/s over REST and 35.8 tok/s in a warm CLI session. **Machine B** (a 3-column aie_next part) — model load, with the current packer: | Weight phase | Range | |---|---| | Packing (no cache) | 3.50 – 13.51 s, median 8.76 | | Cache hit | 0.07 – 3.57 s, median 3.56 | First load additionally writes a 2.0 GB cache in ~1–2 s. The 50× spread within cache hits is page-cache warmth. Load on this machine varies about 4× run to run on an otherwise idle box, so these are ranges rather than single figures — a single sample of this phase is not meaningful. `docs/docs/benchmarks/phi4_results.md` keeps the two machines separate and says plainly which figures were measured where, and which were not re-measured after the packer changed. --- 1. **Greedy decoding is not reproducible across loads.** Eight loads of the same model answering the same prompt at `temperature=0, top_k=1` produced 2 distinct answers; serialising the packer produced 3, so this is **not** caused by the concurrent creates. It is consistent with the requantization refit not being bit-reproducible, and is expected to disappear when dedicated ELFs remove the refit. 2. **The acceptance matrix was re-run in full** after the move to corelib 0.5.0, on an aie_next host: `passed: true`, 0 failures, 10/10 fresh-process CLI load-and-generate cycles, REST both streaming and non-streaming, cancellation clean, the capacity boundary rejected before submission. Cold TTFT 4.65 s, warm TTFT 35.1 ms, decode 34.6 tok/s. The three unit suites are 11/11 on the same box, with `test_real_corelib` running against the real DLL rather than skipping. 3. **`calculate_file_sha256` is ~8× slower than the work requires** — ~28 s over 4 GB where `Get-FileHash` takes 3.67 s, because it uses a portable pure-C++ SHA-256. No longer on the startup path, but every `flm pull` and `flm check` pays it, for every model. Shared pull code, unrelated to this backend, left for its own change. 4. **A shared `FLM_RAI_WEIGHT_CACHE` directory holds one cache**, so alternating two models there repacks each time. The default — beside the model — does not have this. 5. **No packaging.** MSI, WiX and Inno are untouched; this is a rai-enabled developer build. 6. **No CI** builds or runs any of this, and the release preset does not compile the rai path. 7. **Windows only, `Q8_0` only** — no Q4_0/Q4_K/Q6_K, no mixed quantization, no generic GGUF runtime, no other model family. 8. **Usable context is 4095**, against 128k for the model and 32k for the stx entry. ROCm/FastFlowLM#706 reaches the same backend through an **ONNX manifest + packed weights** pipeline and carries packaging this branch does not. This is the **GGUF-direct** approach: smaller surface, no manifest generator, no converted artifacts, no Python at runtime. They are alternatives; only one should land. --------- Co-authored-by: Claude Opus 5 (1M context) Co-authored-by: alfxu_amdeng --- docs/docs/benchmarks/phi4_results.md | 172 ++++ docs/docs/instructions/cli.md | 34 + docs/docs/models/phi.md | 65 ++ src/CMakeLists.txt | 40 + src/CMakePresets.json | 20 + src/common/AutoModel/automodel.cpp | 292 +++++- src/common/AutoModel/builtin_backends.cpp | 62 ++ src/common/AutoModel/model_backend.cpp | 147 +++ src/common/AutoModel/modeling_gemma3.cpp | 14 +- src/common/AutoModel/modeling_gemma3_text.cpp | 14 +- src/common/AutoModel/modeling_gemma4_12b.cpp | 21 +- .../AutoModel/modeling_gemma4_12b_image.cpp | 2 +- src/common/AutoModel/modeling_gemma4e.cpp | 24 +- src/common/AutoModel/modeling_gpt_oss.cpp | 10 +- src/common/AutoModel/modeling_hunyuan.cpp | 18 +- src/common/AutoModel/modeling_lfm2.cpp | 30 +- src/common/AutoModel/modeling_llama3.cpp | 31 +- src/common/AutoModel/modeling_nanbeige.cpp | 17 +- src/common/AutoModel/modeling_phi4.cpp | 233 +++-- src/common/AutoModel/modeling_qwen2.cpp | 15 +- src/common/AutoModel/modeling_qwen2vl.cpp | 13 +- src/common/AutoModel/modeling_qwen3.cpp | 64 +- .../AutoModel/modeling_qwen3_5_omni.cpp | 2 +- src/common/AutoModel/modeling_qwen3_5vl.cpp | 19 +- .../AutoModel/modeling_qwen3_5vl_image.cpp | 2 +- src/common/AutoModel/modeling_qwen3_6_moe.cpp | 19 +- .../AutoModel/modeling_qwen3_6_moe_image.cpp | 2 +- src/common/AutoModel/modeling_qwen3vl.cpp | 21 +- src/common/models/README.md | 347 +++++++ src/common/models/models_sources.cmake | 7 + src/common/models/phi4/rai/phi4_rai.cpp | 495 ++++++++++ .../models/phi4/rai/phi4_rai_backend.cpp | 131 +++ src/common/models/phi4/rai/phi4_rai_gguf.cpp | 658 +++++++++++++ src/common/models/phi4/rai/phi4_rai_host.cpp | 177 ++++ .../models/phi4/rai/phi4_rai_shape_plan.cpp | 117 +++ .../models/phi4/rai/phi4_rai_weight_cache.cpp | 173 ++++ src/common/npu_platform.cpp | 13 + src/common/rai/corelib_api.cpp | 206 +++++ src/common/rai/corelib_runtime.cpp | 110 +++ src/common/rai/rai_sources.cmake | 6 + src/common/tokenizer/tokenizer.cpp | 3 + src/include/AutoModel/automodel.hpp | 85 +- src/include/AutoModel/flm_backend.hpp | 64 ++ src/include/AutoModel/model_backend.hpp | 207 +++++ src/include/AutoModel/modeling_gemma3.hpp | 2 +- .../AutoModel/modeling_gemma3_text.hpp | 2 +- src/include/AutoModel/modeling_gemma4_12b.hpp | 2 +- src/include/AutoModel/modeling_gemma4e.hpp | 13 +- src/include/AutoModel/modeling_gpt_oss.hpp | 2 +- src/include/AutoModel/modeling_hunyuan.hpp | 2 +- src/include/AutoModel/modeling_lfm2.hpp | 4 +- src/include/AutoModel/modeling_llama3.hpp | 4 +- src/include/AutoModel/modeling_nanbeige.hpp | 2 +- src/include/AutoModel/modeling_phi4.hpp | 64 +- src/include/AutoModel/modeling_qwen2.hpp | 2 +- src/include/AutoModel/modeling_qwen2vl.hpp | 2 +- src/include/AutoModel/modeling_qwen3.hpp | 8 +- .../AutoModel/modeling_qwen3_5_omni.hpp | 2 +- src/include/AutoModel/modeling_qwen3_5vl.hpp | 2 +- .../AutoModel/modeling_qwen3_6_moe.hpp | 2 +- src/include/AutoModel/modeling_qwen3vl.hpp | 17 +- src/include/lm_config.hpp | 5 +- src/include/model_list.hpp | 587 +++++++----- src/include/models/phi4/rai/phi4_rai.hpp | 44 + .../models/phi4/rai/phi4_rai_backend.hpp | 38 + .../models/phi4/rai/phi4_rai_constants.hpp | 44 + src/include/models/phi4/rai/phi4_rai_gguf.hpp | 80 ++ src/include/models/phi4/rai/phi4_rai_host.hpp | 35 + .../models/phi4/rai/phi4_rai_shape_plan.hpp | 45 + .../models/phi4/rai/phi4_rai_weight_cache.hpp | 96 ++ src/include/program_args.hpp | 1 + src/include/rai/corelib_api.hpp | 116 +++ src/include/rai/corelib_device.hpp | 25 + src/include/rai/corelib_object.hpp | 62 ++ src/include/rai/corelib_runtime.hpp | 38 + src/include/utils/debug_utils.hpp | 14 + src/include/utils/file_access.hpp | 27 + src/include/utils/npu_platform.hpp | 58 ++ src/include/utils/utils.hpp | 4 + src/include/utils/vm_args.hpp | 3 + src/model_info.json | 26 + src/model_list.json | 83 +- src/pull/download_model.cpp | 152 ++- src/pull/download_model.hpp | 17 + src/pull/model_downloader.cpp | 365 +++++--- src/pull/model_downloader.hpp | 14 +- src/runner/runner.cpp | 27 +- src/runner/runner.hpp | 10 +- src/server/rest_handler.cpp | 178 ++-- src/server/rest_handler.hpp | 284 +++--- src/server/server.cpp | 109 +-- src/server/server.hpp | 93 +- src/src/benchmarking.hpp | 7 +- src/src/main.cpp | 120 ++- src/test/model_backend/CMakeLists.txt | 53 ++ src/test/model_backend/test_model_backend.cpp | 261 ++++++ src/test/model_list_platform/CMakeLists.txt | 20 + .../test_model_list_platform.cpp | 211 +++++ src/test/phi4_rai/CMakeLists.txt | 264 ++++++ src/test/phi4_rai/fake_corelib.cpp | 596 ++++++++++++ src/test/phi4_rai/fake_corelib.hpp | 150 +++ src/test/phi4_rai/gguf_fixture.hpp | 415 +++++++++ src/test/phi4_rai/run_real_rai_acceptance.ps1 | 232 +++++ src/test/phi4_rai/test_corelib_api.cpp | 307 +++++++ src/test/phi4_rai/test_model_downloader.cpp | 412 +++++++++ src/test/phi4_rai/test_phi4_engine.cpp | 742 +++++++++++++++ src/test/phi4_rai/test_phi4_frontend.cpp | 868 ++++++++++++++++++ src/test/phi4_rai/test_phi4_gguf.cpp | 580 ++++++++++++ src/test/phi4_rai/test_phi4_host.cpp | 185 ++++ src/test/phi4_rai/test_phi4_shape_plan.cpp | 123 +++ src/test/phi4_rai/test_real_corelib.cpp | 38 + src/test/phi4_rai/test_support.hpp | 61 ++ 112 files changed, 11503 insertions(+), 1156 deletions(-) create mode 100644 src/common/AutoModel/builtin_backends.cpp create mode 100644 src/common/AutoModel/model_backend.cpp create mode 100644 src/common/models/README.md create mode 100644 src/common/models/models_sources.cmake create mode 100644 src/common/models/phi4/rai/phi4_rai.cpp create mode 100644 src/common/models/phi4/rai/phi4_rai_backend.cpp create mode 100644 src/common/models/phi4/rai/phi4_rai_gguf.cpp create mode 100644 src/common/models/phi4/rai/phi4_rai_host.cpp create mode 100644 src/common/models/phi4/rai/phi4_rai_shape_plan.cpp create mode 100644 src/common/models/phi4/rai/phi4_rai_weight_cache.cpp create mode 100644 src/common/npu_platform.cpp create mode 100644 src/common/rai/corelib_api.cpp create mode 100644 src/common/rai/corelib_runtime.cpp create mode 100644 src/common/rai/rai_sources.cmake create mode 100644 src/include/AutoModel/flm_backend.hpp create mode 100644 src/include/AutoModel/model_backend.hpp create mode 100644 src/include/models/phi4/rai/phi4_rai.hpp create mode 100644 src/include/models/phi4/rai/phi4_rai_backend.hpp create mode 100644 src/include/models/phi4/rai/phi4_rai_constants.hpp create mode 100644 src/include/models/phi4/rai/phi4_rai_gguf.hpp create mode 100644 src/include/models/phi4/rai/phi4_rai_host.hpp create mode 100644 src/include/models/phi4/rai/phi4_rai_shape_plan.hpp create mode 100644 src/include/models/phi4/rai/phi4_rai_weight_cache.hpp create mode 100644 src/include/rai/corelib_api.hpp create mode 100644 src/include/rai/corelib_device.hpp create mode 100644 src/include/rai/corelib_object.hpp create mode 100644 src/include/rai/corelib_runtime.hpp create mode 100644 src/include/utils/file_access.hpp create mode 100644 src/include/utils/npu_platform.hpp create mode 100644 src/test/model_backend/CMakeLists.txt create mode 100644 src/test/model_backend/test_model_backend.cpp create mode 100644 src/test/model_list_platform/CMakeLists.txt create mode 100644 src/test/model_list_platform/test_model_list_platform.cpp create mode 100644 src/test/phi4_rai/CMakeLists.txt create mode 100644 src/test/phi4_rai/fake_corelib.cpp create mode 100644 src/test/phi4_rai/fake_corelib.hpp create mode 100644 src/test/phi4_rai/gguf_fixture.hpp create mode 100644 src/test/phi4_rai/run_real_rai_acceptance.ps1 create mode 100644 src/test/phi4_rai/test_corelib_api.cpp create mode 100644 src/test/phi4_rai/test_model_downloader.cpp create mode 100644 src/test/phi4_rai/test_phi4_engine.cpp create mode 100644 src/test/phi4_rai/test_phi4_frontend.cpp create mode 100644 src/test/phi4_rai/test_phi4_gguf.cpp create mode 100644 src/test/phi4_rai/test_phi4_host.cpp create mode 100644 src/test/phi4_rai/test_phi4_shape_plan.cpp create mode 100644 src/test/phi4_rai/test_real_corelib.cpp create mode 100644 src/test/phi4_rai/test_support.hpp diff --git a/docs/docs/benchmarks/phi4_results.md b/docs/docs/benchmarks/phi4_results.md index 7d7de5ea8..bd00111a2 100644 --- a/docs/docs/benchmarks/phi4_results.md +++ b/docs/docs/benchmarks/phi4_results.md @@ -41,3 +41,175 @@ AMD Ryzen™ AI 7 350 (Kraken Point) with 32 GB DRAM; performance is comparable | **Model** | **HW** | **1k** | **2k** | **4k** | **8k** | **16k** | **32k** | |------------------|--------------------|--------:|--------:|--------:|--------:|---------:|---------:| | **Phi-4-mini-instruct** | NPU (FLM) | 643 | 787 | 857 | 809 | 644 | 447 | + +--- + +## 🧪 Phi-4-mini-instruct Q8_0 GGUF on the rai backend (`phi4-mini-it:4b`, resolved for `aie_next`) + +These are **descriptive measurements from individual acceptance runs**, not a benchmark sweep and not a pass threshold. Each figure below comes from one run, not from an average over many. They are not comparable to the tables above: the prompts here are 4–10 tokens, whereas those tables sweep 1k–32k, so the per-token rates are dominated by fixed overhead rather than by context length. + +## Machine A + +Two runs from this machine are reported. The **current** one is the full +acceptance matrix at the restructured tip, with the concurrent packer and the +on-disk weight cache both in play. The **earlier** one predates both; it is kept +because its load profiling is what the 45 s → 5 s section explains, and because +it is still the only run that measured the serial packer. + +### Current run + +Full acceptance matrix, `passed: true`, 0 failures. + +#### Provenance + +| | | +|---|---| +| Machine | aie_next development machine A | +| CPU | AMD Ryzen AI engineering sample | +| NPU | `AMD XDNA(TM) NPU` | +| OS | Microsoft Windows 11 Enterprise 10.0.26100 build 26100 | +| FastFlowLM commit | `b5a05ac2c798518a0969858bebe2ef5c7f1638eb` (plus two test-harness fixes that do not touch shipping code) | +| corelib commit / ABI | `3c35aebdefa3f0c2255668bab1be5648ece320f8` / `0.3.0` | +| corelib DLL SHA-256 | `f404da219a3cc84d3334c265e09ba7987f0c4bcc1b1cedeac7c3c45a7be2c9ae` | +| corelib actually loaded | confirmed from the live server's loaded-module list: exactly one `ryzenai_corelib.dll`, and it is the file above | +| GGUF revision | `78eb92a46fc37e6b524df991ed9aca9bc6aa7b80` | +| Tokenizer/config revision | `cfbefacb99257ffa30c83adab238a50856ac3083` | +| Run | 2026-09-17 14:44:00 → 14:53:14 | + +#### Measurements + +The weight cache was already populated when this run started, so the load +figures are **cache-hit** loads: the packed weights are mapped from disk and +nothing is requantized. + +| Metric | Value | Conditions | +|---|---|---| +| Model load, cache warm | **1.52 s** median (1.46 – 1.69 s, 11 loads) | fresh CLI processes, `Model loaded in` as reported by FLM | +| Cold TTFT | **3.92 s** | first prompt in a fresh process | +| Warm TTFT | **43.3 ms** | subsequent prompts in the same process | +| Decode, REST | **17.3 tok/s** | `/api/chat`, 16 generated tokens | + +Cache-warm load is both faster and far tighter than the packing path machine B +measured (3.50 – 13.51 s, a ~4× spread): 11 loads inside a 0.23 s band. That is +the point of the cache — it replaces a variable cost with a fixed one. + +The decode and TTFT figures are single observations on 4–10 token prompts and +carry the same caveats as the earlier run below. The REST decode figure moved +from 21.3 to 17.3 tok/s between the two runs; nothing measured here explains +that, and it is within the noise this document already warns about. + +#### Functional results + +- CLI — 10/10 fresh-process load-and-generate cycles exited 0, `Backend: rai` in every one. +- REST — `/api/chat` and `/v1/chat/completions`, streaming and non-streaming, all 200. +- Cancellation — an in-flight stream cancelled cleanly; the next request returned 200 on the same server. +- Capacity boundary — 4095 admitted, 4096 rejected with **HTTP 400** before submission. +- The four pinned model files are byte-identical to the earlier run. The two weight-cache files sit alongside them in the model directory and are excluded from that check by name. + +### Earlier run + +Measured at the commit named below, which predates the backend restructure, the +concurrent packer and the weight cache. Its load figures describe the **serial** +packer. + +#### Provenance + +| | | +|---|---| +| Machine | aie_next development machine A | +| CPU | AMD Ryzen AI engineering sample | +| NPU | `AMD XDNA(TM) NPU` | +| OS | Microsoft Windows 11 Enterprise 10.0.26100 build 26100 | +| Windows power scheme | Balanced (`381b4222-f694-41f0-9685-ff5bb260df2e`). The NPU power mode is separately set to `performance` by FLM at startup. | +| FastFlowLM commit | `87721089097396579ec4529f50616a6c0e1c7b74` | +| corelib commit / ABI | `3c35aebdefa3f0c2255668bab1be5648ece320f8` / `0.3.0` | +| corelib DLL SHA-256 | `f404da219a3cc84d3334c265e09ba7987f0c4bcc1b1cedeac7c3c45a7be2c9ae` | +| GGUF revision | `78eb92a46fc37e6b524df991ed9aca9bc6aa7b80` | +| Tokenizer/config revision | `cfbefacb99257ffa30c83adab238a50856ac3083` | +| Run | 2026-09-12 00:34:19 → 00:52:02, `passed: true`, 0 failures | + +#### Measurements + +| Metric | Value | Conditions | +|---|---|---| +| Model load to serving | **5.1 / 5.3 s** | fresh `flm serve` processes, timed from launch to the first successful `/api/version`. Was 44–49 s at the accepted commit; see below. | +| Cold TTFT | **4.21 s** | first prompt in a fresh process; includes one-time kernel and ELF setup | +| Warm TTFT | **65.0 ms** | subsequent prompts in the same process | +| Decode, REST | **21.3 tok/s** | `/api/chat`, 16 generated tokens | +| Decode, warm CLI session | **35.8 tok/s** | 10 prompts in one loaded process | + +#### Startup: 45 s → 5 s + +The acceptance run measured 44–49 s to serving. Profiling it with `FLM_RAI_PROFILE_LOAD=1` found two independent costs, both since fixed: + +| Phase | Before | After | +|---|---|---| +| Startup integrity check — SHA-256 over the 4 GB GGUF | ~28 s (62%) | **0 s** — not run | +| Weight requantization — 161 objects from Q8_0 | 15.3 / 14.9 s | **2.5 / 3.0 s** | +| Shape plan | 0.05 s | 0.05 s | +| GGUF resolve, host prep, device tensors | < 0.2 s | < 0.2 s | +| **Process launch to serving** | **44.9 / 46.3 s** | **5.1 / 5.3 s** | + +The integrity check was re-hashing every pinned file on every launch — a pull-time concern on the startup path. `flm pull` and `flm check` still verify in full; only the run and serve paths were changed to ask for status alone. + +The packer was being given a threads hint of 0, which corelib treats as ONE deliberately, so a single create packed on a single thread. Raising the hint brought requantization to 2.5–3.0 s here, within range of the 2.2 s `python/phi4_driver.py` reports for the same 161 weights. The packer has since moved to concurrent creates instead; machine B carries those figures. + +Output was re-verified after the change: `2+2` → `4`, `capital of France` → `Paris`, `primary color` → `Red.`, and a correct one-sentence description of AMD. + +Separately, and **not** fixed: `calculate_file_sha256` uses a portable pure-C++ SHA-256 with no hardware acceleration, and takes ~28 s over 4 GB where `Get-FileHash` on the same machine takes **3.67 s**. That ~8× gap is not specific to this model or backend — it is still paid by `flm pull` and `flm check` for every model. + +**Do not read the per-process cold cycles as throughput.** Ten fresh-process cycles generating 8 tokens each reported 3.70–20.26 tok/s decode and 1.09–3.65 tok/s prefill. Every one of those pays the one-time setup inside its own measurement window, so the average describes start-up cost, not steady-state speed. + +The **5.4×** spread between warm TTFT (65 ms) and cold TTFT (4.21 s), and the **1.7×** spread between the REST and warm-CLI decode figures, are both unexplained by anything measured here. Treat single-run differences below roughly 2× as noise. + +#### Functional results + +All from the same run: + +- `flm pull` / `flm check` — four pinned files, all SHA-256 verified; the model directory contains exactly those four. +- CLI — 10/10 fresh-process load-and-generate cycles exited 0; the backend id and the loaded DLL path were reported in every one. The id that run printed was an earlier name for what is now `rai`, so it is described rather than quoted here. +- REST — `/api/chat` and `/v1/chat/completions` both 200, streaming and non-streaming. +- Cancellation — an in-flight stream cancelled cleanly; the next request returned 200 on the same server. +- Capacity boundary — a request totalling 4096 tokens is rejected with **HTTP 400** before submission (`rendered prompt has 4 tokens and requested output has 4092 tokens`); a 4095-token request is admitted. +- No CPU or NPU2 fallback appears in the server log at any point. + +#### Known issue + +One `/api/chat` reply to `What is 2+2?` came back as a truncated markdown image URL (`![](https://media.giphy.com/media/kZl76FZgu`, `done_reason: length`) instead of an answer. The identical prompt answered correctly on three other occasions in the same session, including the recovery request in the same run, so this looks like sampling nondeterminism rather than a routing fault — but it is a single-observation defect, it is not understood, and it is recorded rather than smoothed over. + +--- + +## Machine B + +Measured with the concurrent packer, after the backend restructure. **Load only**: TTFT, decode throughput and the acceptance matrix have not been re-run on this machine, so machine A remains the only source for those. + +### Provenance + +| | | +|---|---| +| Machine | aie_next development machine B | +| NPU | architecture `aie_next` | +| CPU | AMD Ryzen AI engineering sample, 20 cores | +| Memory | 32 GB | +| OS | Windows 11 Enterprise 10.0.26100.4652 | +| XRT / NPU driver / NPU firmware | 2.25.0 / 32.0.20214.4161 / 2.6.1.219 | +| corelib | `3c35aebd`, ABI 0.3.0, built on this machine | + +### Model load + +Weight requantization is effectively the whole of load; everything else — shape planning, GGUF resolution, host preparation, device allocation — stays under a quarter of a second combined. + +| Packer | Requantization | Notes | +|---|---|---| +| Serial, one create at a time | **~30 s** | 29.99 / 30.19 / 25.14 s across runs, consistently slow | +| Concurrent, 8 creates in flight | **3.50 – 13.51 s**, median 8.76 s over 6 runs | one interactive run measured 7.37 s | + +Two things are worth separating here. + +The **consistent** 30 s came from the packer running effectively single-threaded in that session while the same binary was several times faster elsewhere. Taking the parallelism as threads FastFlowLM owns, rather than as a hint passed to the packer, removes that dependence on the surrounding environment. + +What remains is **variance, not a fixed cost**: 3.50–13.51 s on an otherwise idle machine, a ~4× spread, and a separate ten-load run saw 6.74–19.41 s. Single measurements of this phase are not meaningful; quote a range. The variance is not explained by anything measured here. + +### Not measured here + +Load is where this machine was exercised. Cold and warm TTFT, decode throughput, the REST and cancellation matrix, and the capacity boundary were all measured on machine A at an earlier commit and have **not** been reconfirmed here. diff --git a/docs/docs/instructions/cli.md b/docs/docs/instructions/cli.md index f31d81203..e8cc76a76 100644 --- a/docs/docs/instructions/cli.md +++ b/docs/docs/instructions/cli.md @@ -232,6 +232,40 @@ flm serve llama3.2:1b --ctx-len 8192 --- +### 🔀 Choose an Execution Backend + +**A backend names where the kernels come from.** There are two: + +| id | kernels | ships on | +|---|---|---| +| `flm` | FastFlowLM's own NPU kernels | Strix / Krackan Point | +| `rai` | AMD's ryzenai-corelib | the next NPU generation | + +You normally never set this. A build links one kernel flow — `FLM_ENABLE_RAI` selects `rai`, otherwise `flm` — and that is the default. `flm run` separately prints the silicon it was built for as `NPU platform: stx`. A model family has at most one engine per backend, so there is nothing to choose between. + +The flag exists for overriding the detection, and for the targets that will join this list later: + +```shell +flm run phi4-mini-it:4b --backend flm +flm serve phi4-mini-it:4b --backend rai +``` + +**Precedence**, highest first: + +| # | Source | | +|---|---|---| +| 1 | `--backend ` | the flag above, or a `"backend"` field on an `/api/chat` or `/api/generate` request | +| 2 | `FLM_BACKEND=` | environment variable, for a whole shell session | +| 3 | the detected NPU | what `flm validate` reports | + +A per-request `"backend"` overrides `--backend` for that request, and reloads the model if it differs from the one already loaded — exactly as asking for a different model does. + +Asking for a backend this build does not ship fails at load with a message listing what is actually available. There is no silent fallback to another engine or to the CPU. Note that a release is built for one NPU generation: on hardware it was not built for, `run`, `serve` and `bench` refuse up front, and `flm validate` still reports what it found. + +> A build carries the engines for one generation only, so there is no flag that moves it to the other catalog entry — that is a different build of FLM. + +--- + ### 🖧 Set Server Port at Launch Set a custom port at launch: diff --git a/docs/docs/models/phi.md b/docs/docs/models/phi.md index 9b2c59bff..e838af7a3 100644 --- a/docs/docs/models/phi.md +++ b/docs/docs/models/phi.md @@ -22,4 +22,69 @@ parent: Models flm run phi4-mini-it:4b ``` +--- + +## 🧪 Model Card: Phi-4-mini-instruct on the rai backend (developer preview) + +- **Tag:** `phi4-mini-it:4b` — the same tag as the NPU2 build. A build targets one NPU generation (`FLM_ENABLE_RAI` selects `aie_next`, otherwise `stx` / Strix / Krackan Point), and the tag resolves to the artifacts that generation can run. There is no separate tag for it; `flm list` on a rai build shows only the models it can run. +- **Backend:** `rai` — the backend names the kernel provider; FastFlowLM reaches these kernels through AMD's `ryzenai_corelib` +- **Source format:** GGUF, read directly. No ONNX model, no tensor manifest, and no converted or packed weight file is produced or shipped. +- **Quantization:** GGML `Q8_0` in the file, requantized to **group 64** while the weights are packed for the device, through corelib's explicit `*_create_gguf_requantized` entry points. This is a **lossy** second quantization step and it is not reversible; output will differ from the Q8_0 source. +- **Usable generation window:** 4095 tokens — the rendered prompt plus the requested output together, so the largest admissible prompt is 4094. An over-capacity request is rejected with HTTP 400 *before* any work is submitted to the device. Note this is far below the model's 128k context; see below for why. +- **Availability:** Windows only, and this is a **developer build**. The rai runtime is not packaged by the MSI or Inno installer; you build against corelib yourself. + +On rai this tag pulls from two pinned repositories, because the GGUF publisher does not ship the tokenizer files FastFlowLM's tokenizer frontend consumes: + +| File | Repository | Revision | +|---|---|---| +| `Phi-4-mini-instruct.Q8_0.gguf` | [`unsloth/Phi-4-mini-instruct-GGUF`](https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF) | `78eb92a46fc37e6b524df991ed9aca9bc6aa7b80` | +| `tokenizer.json` | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) | `cfbefacb99257ffa30c83adab238a50856ac3083` | +| `tokenizer_config.json` | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) | `cfbefacb99257ffa30c83adab238a50856ac3083` | +| `config.json` | [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct) | `cfbefacb99257ffa30c83adab238a50856ac3083` | + +All four are SHA-256 verified before the download is promoted, and the pulled directory contains exactly these four files. + +### Building + +The rai path is compiled only when you ask for it. With the option off, the binary contains no reference to corelib at all. + +From `FastFlowLM/src`, in a Visual Studio developer command prompt: + +```powershell +$env:RYZENAI_CORELIB_INCLUDE_DIR = 'C:/path/to/ryzenai-corelib/install/include' +cmake --preset windows-rai # sets FLM_ENABLE_RAI=ON, builds into src/build-rai +cmake --build --preset windows-rai +``` + +The `windows-rai` preset reads `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY` from the environment, so set both before configuring. The configure step also locates a Boost include directory, and hard-errors if the option is enabled on a non-Windows host. Everything else — XRT, FFmpeg, curl, FFTW — is the ordinary FastFlowLM dependency set; the rai option does not relax any of it. + +### Pointing FastFlowLM at the runtime + +A rai build (`-DFLM_ENABLE_RAI=ON`) **links corelib in**, because the NPU device the whole process shares comes from corelib's `ryzenai::corelib::GetDevice()` rather than from a device FastFlowLM opens itself. Point the build at the library with `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY`. `FLM_RAI_CORELIB_PATH` selects a DLL only in the older dynamically loading configuration; in a statically linked rai build it is ignored, and `flm` says so if it is set. + +The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.3.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. Corelib's own dependency directory must be reachable on `PATH`. + +```powershell +flm pull phi4-mini-it:4b +flm run phi4-mini-it:4b +``` + +### Why the context is 4096, and why the usable window is one less + +Phi-4-mini itself supports 128k, and the existing `phi4-mini-it:4b` tag defaults to 32k. This backend gives you 4095. That is a real functional regression and it has two separate causes, which are worth keeping apart. + +**The 4096 ceiling is a correctness boundary, not a buffer size.** 4096 is exactly Phi-4-mini's `rope.scaling.original_context_length`. LongRoPE selects its factors by *sequence length*, not per position: at or below the original length the short factors apply, above it the long ones do. This implementation derives only the short branch, so 4096 is the point past which the rope tables would silently be wrong. It is enforced rather than assumed — loading fails with `invalid Phi-4 RoPE metadata` unless the GGUF reports `rope.scaling.original_context_length` of exactly 4096. Raising this ceiling means deriving the long factors, not enlarging an array. + +**The extra −1 is this frontend's own conservatism.** `kMaxDecodeWindow` is 4095, one below the attention window, so that any request the server admits is guaranteed to have room to finish rather than failing partway. It costs exactly one token and it is not imposed by corelib. + +### No fallback + +Backend selection follows the build, never a filename or a quantization level. What this binary was built for decides two things: *which catalog entry* the tag resolves to — the NPU2/Q4NX entry on `stx`, this one on `aie_next` — and which backend runs it, since a build links exactly one kernel flow. Once this entry is selected, there is no fallback: if corelib is missing, unloadable, or the wrong version, the tag **fails to load with a diagnostic** rather than quietly running on CPU or on the NPU2/Q4NX backend. + +### Naming the backend yourself + +The two engines are registered under the kernel provider they use, `flm` and `rai`, and you can name one with `--backend`, with `FLM_BACKEND`, or with a `"backend"` field on an `/api/chat` or `/api/generate` request. The [CLI reference](../instructions/cli.md) has the full precedence table. + +This does not widen what the hardware can run. A given release is built for one NPU generation, so only that generation's backend is compiled in; asking for the other one fails immediately, naming what this build actually has, instead of failing deep inside an engine that was never going to work. If what you meant was the other catalog entry, that is a different build of FLM, not a different flag. + --- \ No newline at end of file diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 588149974..254fd0564 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -48,6 +48,27 @@ set(CMAKE_RUNTIME_OUTPUT_DIRECTORY_RELEASE ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}) # ——————————————————————————————————————————————— option(FLM_USE_HRX "Use the HRX amdxdna NPU runtime instead of XRT (0=XRT default, 1=HRX)" OFF) option(FLM_PORTABLE_BUILD "Build portable distribution with bundled runtime libraries" OFF) +# Where the kernels come from. FastFlowLM's own flow and ryzenai-corelib share +# no engine, so a build links one or the other; this option is not an extra. +# It also fixes the NPU generation the build targets -- see utils/npu_platform.hpp. +option(FLM_ENABLE_RAI + "Reach kernels through ryzenai-corelib instead of FastFlowLM's own flow" OFF) + +if(FLM_ENABLE_RAI) + if(NOT WIN32) + message(FATAL_ERROR "FLM_ENABLE_RAI currently requires Windows") + endif() + find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h REQUIRED) + # The main binary needs ryzenai::corelib::GetDevice(), a C++ entry point the + # C ABI does not expose, so corelib is linked in rather than dlopened. + find_library(RYZENAI_CORELIB_LIBRARY NAMES ryzenai_corelib corelib + HINTS "${RYZENAI_CORELIB_INCLUDE_DIR}/../lib" + "$ENV{RYZENAI_CORELIB_LIB_DIR}" REQUIRED) + find_path(FLM_CORELIB_BOOST_INCLUDE_DIR NAMES boost/any.hpp + HINTS "$ENV{CONDA_PREFIX}/Library/include" + "$ENV{USERPROFILE}/anaconda3/Library/include" + "C:/dev/boost_1_88_0" REQUIRED) +endif() if(FLM_USE_HRX) set(FLM_RUNTIME_NAME "hrx") @@ -237,6 +258,10 @@ add_subdirectory(${CMAKE_SOURCE_DIR}/../third_party/tokenizers-cpp # ——————————————————————————————————————————————— file(GLOB SOURCES "src/*.cpp" "runner/*.cpp" "common/*.cpp" "common/*/*.cpp" "server/*.cpp" "pull/*.cpp" ) file(GLOB HEADERS "include/*.hpp" "runner/*.hpp" "common/*.hpp" "common/*/*.hpp" "server/*.hpp" "pull/*.hpp") +list(FILTER SOURCES EXCLUDE REGEX ".*/common/rai/.*\\.cpp$") + +# Model sources live two levels deeper than the globs above reach. Only the rai +# half exists as loose sources; it is built separately, below. # Exclude files that depend on missing libraries for Linux if(NOT WIN32) @@ -269,6 +294,21 @@ endif() add_executable(flm ${SOURCES} ${HEADERS}) +if(FLM_ENABLE_RAI) + include("${CMAKE_SOURCE_DIR}/common/rai/rai_sources.cmake") + add_library(flm_rai STATIC ${FLM_RAI_SOURCES}) + target_include_directories(flm_rai PUBLIC + "${CMAKE_SOURCE_DIR}/include" "${RYZENAI_CORELIB_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}" "${FLM_CORELIB_BOOST_INCLUDE_DIR}") + # RYZENAI_CORELIB_STATIC drops the vendor header's dllimport decoration; + # FLM_CORELIB_LINK_STATIC is what tells our own code the symbols are linked + # in and must be bound directly instead of through LoadLibrary. + target_compile_definitions(flm_rai PUBLIC + FLM_ENABLE_RAI=1 RYZENAI_CORELIB_STATIC=1 FLM_CORELIB_LINK_STATIC=1) + target_link_libraries(flm_rai PUBLIC "${RYZENAI_CORELIB_LIBRARY}") + target_link_libraries(flm PRIVATE flm_rai) +endif() + if(WIN32) if(VCPKG_TOOLCHAIN) # A vcpkg toolchain is active (e.g. the rocm-npu-staging dev.py build or diff --git a/src/CMakePresets.json b/src/CMakePresets.json index 2bc1ef119..54fde039f 100644 --- a/src/CMakePresets.json +++ b/src/CMakePresets.json @@ -53,6 +53,18 @@ "CMAKE_BUILD_TYPE": "Release" } }, + { + "name": "windows-rai", + "displayName": "Windows RAI", + "description": "Windows build with statically linked ryzenai-corelib support", + "inherits": "windows-default", + "binaryDir": "${sourceDir}/build-rai", + "cacheVariables": { + "FLM_ENABLE_RAI": "OFF", + "RYZENAI_CORELIB_INCLUDE_DIR": "$env{RYZENAI_CORELIB_INCLUDE_DIR}", + "RYZENAI_CORELIB_LIBRARY": "$env{RYZENAI_CORELIB_LIBRARY}" + } + }, { "name": "windows-vs18", "displayName": "Windows VS18", @@ -85,6 +97,14 @@ "configurePreset": "windows-default", "configuration": "Release", "jobs": 4 + }, + { + "name": "windows-rai", + "displayName": "Windows RAI Build", + "description": "Build the optional ryzenai-corelib runtime in Release", + "configurePreset": "windows-rai", + "configuration": "Release", + "jobs": 4 } ] } diff --git a/src/common/AutoModel/automodel.cpp b/src/common/AutoModel/automodel.cpp index 78a4a0f88..0c821872c 100644 --- a/src/common/AutoModel/automodel.cpp +++ b/src/common/AutoModel/automodel.cpp @@ -6,7 +6,42 @@ /// \note This is a source file for the auto_model class #include "AutoModel/automodel.hpp" +#include "utils/npu_platform.hpp" + +namespace { + +/// \brief the context length a model should actually be loaded with +/// \param model_info the catalog entry +/// \param default_context_length the --ctx-len value, or -1 when not given +/// \return the resolved length +/// \note Single-turn models -- dedicated translation models, say -- do not +/// support an arbitrary override, so they always get their own default. +int resolve_context_length(const nlohmann::ordered_json& model_info, + int default_context_length) { + if (default_context_length == -1) { + return model_info["default_context_length"].get(); + } + const bool single_turn = + model_info.contains("label") && + std::find(model_info["label"].begin(), model_info["label"].end(), + "single-turn") != model_info["label"].end(); + if (single_turn) { + header_print("FLM", "Single-turn model, 1k max context length allowed only!"); + return model_info["default_context_length"].get(); + } + return default_context_length; +} + +} // namespace + + +ModelRequestError::ModelRequestError( + int http_code, bool session_cleared, std::string message) + : std::runtime_error(std::move(message)), + http_code_(http_code), session_cleared_(session_cleared) {} +int ModelRequestError::http_code() const noexcept { return http_code_; } +bool ModelRequestError::session_cleared() const noexcept { return session_cleared_; } AutoModel::AutoModel(flm_rt::device* npu_device_inst, std::string current_model) { this->npu_device_inst = npu_device_inst; @@ -106,16 +141,23 @@ nlohmann::json AutoModel::_shared_setup_tokenizer(std::string model_path) { this->bos_token_id = -1; } this->eos_token = tokenizer_config["eos_token"].get(); - if (!tokenizer_config["eos_token_id"].is_array()) { - header_print("ERROR", "eos_token_id is missing or not an array in tokenizer_config.json"); - exit(1); - } - for (auto& token : tokenizer_config["eos_token_id"]) { - if (!token.is_number_integer()) { - header_print("ERROR", "eos_token_id must be an array of integers in tokenizer_config.json"); + // A backend that cross-validated its own package knows the stop ids better + // than tokenizer_config.json does. + if (const auto forced = this->backend_ ? this->backend_->forced_eos_ids() + : std::nullopt) { + this->eos_token_ids = *forced; + } else { + if (!tokenizer_config["eos_token_id"].is_array()) { + header_print("ERROR", "eos_token_id is missing or not an array in tokenizer_config.json"); exit(1); } - this->eos_token_ids.push_back(token.get()); + for (auto& token : tokenizer_config["eos_token_id"]) { + if (!token.is_number_integer()) { + header_print("ERROR", "eos_token_id must be an array of integers in tokenizer_config.json"); + exit(1); + } + this->eos_token_ids.push_back(token.get()); + } } this->user_system_prompt = ""; this->extra_context["user_system_prompt"] = this->user_system_prompt; @@ -128,44 +170,172 @@ void AutoModel::_shared_load_model(std::string model_path, json model_info, int header_print("FLM", "Model already loaded: " << this->model_path); return; } + const int context_length = + resolve_context_length(model_info, default_context_length); + this->_shared_initialize_model_state( + std::move(model_path), std::move(model_info), context_length); + this->_shared_initialize_legacy_npu(enable_preemption); +} - this->model_path = model_path; +void AutoModel::_shared_initialize_model_state( + std::string model_path, json, int context_length) { + this->model_path = std::move(model_path); header_print("FLM", "Loading model: " << this->model_path); this->lm_config = std::make_unique(); this->lm_config->from_pretrained(this->model_path); - if (this->npu_device_inst == nullptr) { - header_print("ERROR", "NPU device instance is nullptr"); - exit(1); + this->MAX_L = context_length; + this->is_model_loaded = true; + this->token_history.clear(); + this->token_history.reserve(this->MAX_L); + this->tokenizer = std::make_unique(this->model_path); + this->last_token = -1; + this->total_tokens = 0; +} + +void AutoModel::_shared_load_backend(std::string model_path, json model_info, + int default_context_length, + bool enable_preemption, + const std::string& requested_backend, + const nlohmann::json* tokenizer_config) { + if (!model_info.contains("details") || + !model_info["details"].contains("family")) { + throw std::runtime_error("Model entry has no details.family"); + } + const std::string family = model_info["details"]["family"].get(); + + auto& registry = flm::backend::BackendRegistry::instance(); + std::string source; + // A build links one kernel flow, so that is the default when neither + // --backend nor FLM_BACKEND says otherwise. + const std::string id = flm::backend::resolve_backend_id( + family, flm::backend::build_default_backend_id(), requested_backend, + &source); + + // Same model on the same backend is a no-op; a different backend is a real + // reload even when the path has not changed. + if (this->is_model_loaded && this->model_path == model_path && + this->backend_ && this->backend_->id() == id) { + header_print("FLM", "Model already loaded: " << this->model_path); + return; } - this->npu = std::make_unique(npu_device::device_npu2, this->npu_device_inst, enable_preemption); - this->enable_preemption = enable_preemption; - // Single-turn models (e.g. dedicated translation models) don't support arbitrary - // context length overrides, so always fall back to the model's own default. - bool single_turn = model_info.contains("label") && - std::find(model_info["label"].begin(), model_info["label"].end(), "single-turn") != model_info["label"].end(); - // Set context length: use provided value if not -1, otherwise use model default - if (default_context_length != -1 && single_turn) { - header_print("FLM", "Single-turn model, 1k max context length allowed only!"); - this->MAX_L = model_info["default_context_length"]; + + const auto traits = registry.traits(family, id); + if (enable_preemption && !traits.supports_preemption) { + throw std::invalid_argument( + "Backend '" + id + "' does not support preemption"); } - else if (default_context_length != -1) { - this->MAX_L = default_context_length; + const int context_length = + resolve_context_length(model_info, default_context_length); + if (traits.max_context_length != 0 && + (context_length < 1 || + static_cast(context_length) > traits.max_context_length)) { + throw std::out_of_range( + "Backend '" + id + "' context length must be in 1.." + + std::to_string(traits.max_context_length)); } - else { - this->MAX_L = model_info["default_context_length"]; + + // Tear the old model down before building the new one: the engines hold + // device memory, and a half-loaded model must never look loaded. + this->backend_.reset(); + this->lm_engine = nullptr; + this->is_model_loaded = false; + + try { + this->_shared_initialize_model_state( + std::move(model_path), model_info, context_length); + if (traits.needs_npu_xclbin) { + this->_shared_initialize_legacy_npu(enable_preemption); + } else { + this->npu.reset(); + this->enable_preemption = false; + } + + flm::backend::BackendContext context; + context.model_path = this->model_path; + context.model_info = model_info; + context.config = this->lm_config.get(); + context.npu = this->npu.get(); + context.device = this->npu_device_inst; + context.context_length = static_cast(context_length); + context.enable_preemption = this->enable_preemption; + context.tokenizer_config = tokenizer_config; + + this->backend_ = registry.create(family, id, context); + this->lm_engine = &this->backend_->engine(); + } catch (...) { + this->backend_.reset(); + this->lm_engine = nullptr; + this->tokenizer.reset(); + this->sampler.reset(); + this->lm_config.reset(); + this->is_model_loaded = false; + throw; } - - this->is_model_loaded = true; + header_print("FLM", "Backend: " << id << " (from " << source << ")"); +} +void AutoModel::_shared_after_inference_failure(bool poisoned) { + this->total_tokens = 0; + this->last_token = -1; this->token_history.clear(); - this->token_history.reserve(this->MAX_L); - this->tokenizer = std::make_unique(this->model_path); + this->checkpoint_his.clear(); + // A poisoned engine cannot be driven at all, not even to clear itself. + if (!poisoned && this->lm_engine) { + try { this->lm_engine->clear_context(); } catch (...) {} + } + if (this->sampler) this->sampler->reset_penalties(); + this->reset_parser(); +} - this->last_token = -1; - this->total_tokens = 0; +void AutoModel::_shared_guard_poisoned() const { + if (this->backend_ && this->backend_->poisoned()) { + // Same condition as the throw in the frontend's inference-failure path, + // so it says the same thing: one poisoned model must not be described + // two ways depending on whether this is the request that broke it. + throw ModelRequestError(500, true, + "Backend '" + this->backend_->id() + + "' failed; unload/reload is required because the model is poisoned"); + } } -bool AutoModel::_shared_insert(chat_meta_info_t& meta_info, std::vector& tokens, std::function is_cancelled, void* payload, int first_len_run) { +void AutoModel::_shared_initialize_legacy_npu(bool enable_preemption) { + if (this->npu_device_inst == nullptr) { + header_print("ERROR", "NPU device instance is nullptr"); + exit(1); + } + this->npu = std::make_unique( + npu_device::device_npu2, this->npu_device_inst, enable_preemption); + this->enable_preemption = enable_preemption; +} + +std::string AutoModel::generate_with_prompt( + chat_meta_info_t& meta_info, + lm_uniform_input_t& input, + int length_limit, + std::ostream& os, + std::function is_cancelled) { + if (!insert(meta_info, input, is_cancelled)) return {}; + return generate(meta_info, length_limit, os, std::move(is_cancelled)); +} + +void AutoModel::_shared_validate_capacity(std::size_t rendered_tokens, + std::optional requested) const { + if (!this->backend_ || this->backend_->max_decode_length() == 0) return; + const std::size_t cap = this->decode_cap(); + const auto normalized = normalize_requested_max_new_tokens(requested); + if (rendered_tokens >= cap || + (normalized && static_cast(*normalized) > cap - rendered_tokens)) { + std::ostringstream message; + message << "Request exceeds the " << cap + << "-token limit of backend '" << this->backend_->id() + << "': rendered prompt has " << rendered_tokens << " tokens"; + if (normalized) message << " and requested output has " << *normalized << " tokens"; + throw ModelRequestError(400, false, message.str()); + } +} + +bool AutoModel::_shared_insert(chat_meta_info_t& meta_info, std::vector& tokens, std::function is_cancelled, void* payload, int first_len_run, std::optional requested_max_new_tokens) { + this->_shared_validate_capacity(tokens.size(), requested_max_new_tokens); // print token history // header_print("DEBUG", "Current token history: "); @@ -242,6 +412,14 @@ buffer AutoModel::_chunked_insert(chat_meta_info_t& meta_info, std::vector } buffer y; if (max_prefill_len < 512) { + if (is_cancelled()) { + meta_info.stop_reason = CANCEL_DETECTED; + buffer_.clear(); + current_mode_ = StreamEventType::CONTENT; + tool_name_.clear(); + is_in_tool_block_ = false; + return y; + } y = this->lm_engine->prefill(tokens, payload); } else{ @@ -253,19 +431,18 @@ buffer AutoModel::_chunked_insert(chat_meta_info_t& meta_info, std::vector } int chunks = (tokens.size() + max_prefill_len - 1) / max_prefill_len; for (int i = 0; i < chunks; i++) { + int start = i * max_prefill_len; + int end = std::min(static_cast(tokens.size()), (i + 1) * max_prefill_len); + std::vector chunk_tokens(tokens.begin() + start, tokens.begin() + end); + header_print("FLM", "Prefill chunk " + std::to_string(i+1) + "/" + std::to_string(chunks) + " with " + std::to_string(chunk_tokens.size()) + " tokens"); if (is_cancelled()) { meta_info.stop_reason = CANCEL_DETECTED; - // reset stream content buffer_.clear(); current_mode_ = StreamEventType::CONTENT; tool_name_.clear(); is_in_tool_block_ = false; break; } - int start = i * max_prefill_len; - int end = std::min(static_cast(tokens.size()), (i + 1) * max_prefill_len); - std::vector chunk_tokens(tokens.begin() + start, tokens.begin() + end); - header_print("FLM", "Prefill chunk " + std::to_string(i+1) + "/" + std::to_string(chunks) + " with " + std::to_string(chunk_tokens.size()) + " tokens"); buffer chunk_y = this->lm_engine->prefill(chunk_tokens, (i == 0)? payload : nullptr); if (i == chunks - 1) { y = chunk_y; @@ -310,16 +487,28 @@ std::string AutoModel::_shared_generate(chat_meta_info_t& meta_info, int length_ } if (this->is_eos(last_sampled_token)){ + meta_info.stop_reason = reason; return result; } this->profiler_list[DECODING_TIME].reset(); this->profiler_list[TKOEN_DECODE_TIME].reset(); - if (this->total_tokens >= this->MAX_L){ + // Some backends refuse to decode past a limit of their own, below MAX_L. + const uint32_t decode_cap = this->decode_cap(); + if (this->total_tokens >= decode_cap){ header_print("WARNING", "Max length reached, stopping generation..."); reason = MAX_LENGTH_REACHED; + meta_info.stop_reason = reason; return result; } - while (this->total_tokens < this->MAX_L){ + while (true){ + // Running out of window is a truncation, not the model choosing to + // stop, and the caller has to be able to tell those apart: this is + // what becomes `done_reason` over the API. Testing the cap here rather + // than in the loop condition is what records it. + if (this->total_tokens >= decode_cap){ + reason = MAX_LENGTH_REACHED; + break; + } if (is_cancelled()) { reason = CANCEL_DETECTED; // reset stream content @@ -350,7 +539,12 @@ std::string AutoModel::_shared_generate(chat_meta_info_t& meta_info, int length_ this->token_history.push_back(sampled_token); if (this->is_eos(sampled_token)){ meta_info.generated_tokens++; - if (this->forward_on_eos) { + // The FastFlowLM engines want one more forward() to keep their KV + // cache in step. Two things can say no: a frontend whose cache is + // discarded anyway (forward_on_eos), and a backend that caps its + // own decode and would reject the call. + if (this->forward_on_eos && + (!this->backend_ || this->backend_->forwards_past_eos())) { this->lm_engine->forward(last_sampled_token); } break; @@ -363,7 +557,7 @@ std::string AutoModel::_shared_generate(chat_meta_info_t& meta_info, int length_ } meta_info.decoding_duration = (uint64_t)(time_utils::cast_to_us(this->profiler_list[DECODING_TIME].get_total_time()).first) * 1e3; meta_info.stop_reason = reason; - if (this->total_tokens >= this->MAX_L){ + if (this->total_tokens >= decode_cap){ header_print("WARNING", "Max length reached, stopping generation..."); } if (this->log_raw_output) { @@ -504,6 +698,12 @@ void AutoModel::clear_context() { this->last_token = -1; this->token_history.clear(); this->checkpoint_his.clear(); + // A poisoned engine would throw; drop the conversation and leave it alone + // so the caller can still reload. + if (this->backend_ && this->backend_->poisoned()) { + if (this->sampler) this->sampler->reset_penalties(); + return; + } this->lm_engine->clear_context(); this->total_tokens = 0; this->sampler->reset_penalties(); @@ -584,6 +784,14 @@ std::string AutoModel::show_profile() { // ss << " Average token decoding speed: " << this->profiler_list[TKOEN_DECODE_TIME].get_average_speed() << " tokens/s" << std::endl; // ss << " Average overall speed: " << this->profiler_list[TOTAL_TIME].get_average_speed() << " tokens/s" << std::endl; + if (this->backend_) { + ss << " Backend: " << this->backend_->id() << std::endl; + const std::string detail = this->backend_->detail(); + if (!detail.empty()) { + ss << " Backend detail: " << detail << std::endl; + } + } + return ss.str(); } diff --git a/src/common/AutoModel/builtin_backends.cpp b/src/common/AutoModel/builtin_backends.cpp new file mode 100644 index 000000000..0fef482d9 --- /dev/null +++ b/src/common/AutoModel/builtin_backends.cpp @@ -0,0 +1,62 @@ +/// \file builtin_backends.cpp +/// \brief The backends this build ships, keyed by model family +/// \note One place knows both the family names from model_list.json and the +/// engine types behind them. Everything else goes through the registry. +/// \note Every family registers the flm backend. Only families with a corelib +/// engine also register a rai backend, and only when this build has one. +#include "AutoModel/automodel.hpp" +#include "AutoModel/flm_backend.hpp" +#include "AutoModel/model_backend.hpp" + +#if defined(FLM_ENABLE_RAI) +#include "models/phi4/rai/phi4_rai_backend.hpp" +#endif + +namespace flm::backend { +namespace { + +/// \brief register the flm backend for one family +/// \tparam Engine the concrete engine type +/// \param registry the registry to populate +/// \param family the family name, as in details.family +template +void RegisterFlm(BackendRegistry& registry, const char* family) { + registry.register_backend(family, kFlmBackendId, + flm_factory()); +} + +} // namespace + +void register_builtin_backends(BackendRegistry& registry) { + RegisterFlm(registry, "llama3"); + RegisterFlm(registry, "deepseek-r1"); + RegisterFlm(registry, "deepseek-r1-0528"); + RegisterFlm(registry, "qwen2"); + RegisterFlm(registry, "qwen2vl"); + RegisterFlm(registry, "qwen3"); + RegisterFlm(registry, "qwen3-it"); + RegisterFlm(registry, "qwen3-tk"); + RegisterFlm(registry, "qwen3vl"); + RegisterFlm(registry, "qwen3vl-flash"); + RegisterFlm(registry, "qwen3.5"); + RegisterFlm(registry, "qwen3.6-moe"); + RegisterFlm(registry, "gemma3"); + RegisterFlm(registry, "gemma3-text"); + RegisterFlm(registry, "gemma4e"); + RegisterFlm(registry, "gemma4e-flash"); + RegisterFlm(registry, "gemma4-12b"); + RegisterFlm(registry, "hunyuan"); + RegisterFlm(registry, "gpt-oss"); + RegisterFlm(registry, "lfm2"); + RegisterFlm(registry, "lfm2.5-tk"); + RegisterFlm(registry, "nanbeige"); + RegisterFlm(registry, "phi4"); + +#if defined(FLM_ENABLE_RAI) + registry.register_backend("phi4", flm::backend::kRaiBackendId, + flm::phi4::rai_factory(), + flm::phi4::rai_traits()); +#endif +} + +} // namespace flm::backend diff --git a/src/common/AutoModel/model_backend.cpp b/src/common/AutoModel/model_backend.cpp new file mode 100644 index 000000000..62ea33a90 --- /dev/null +++ b/src/common/AutoModel/model_backend.cpp @@ -0,0 +1,147 @@ +/// \file model_backend.cpp +/// \brief The backend registry and the rules for picking a backend +#include "AutoModel/model_backend.hpp" + +#include "utils/utils.hpp" + +#include +#include +#include +#include + +namespace flm::backend { +namespace { + +/// \brief render a list of ids for an error message +/// \param ids the ids +/// \return "a, b, c", or "(none)" when the list is empty +std::string Join(const std::vector& ids) { + if (ids.empty()) return "(none)"; + std::ostringstream out; + for (std::size_t i = 0; i < ids.size(); ++i) { + if (i) out << ", "; + out << ids[i]; + } + return out.str(); +} + +/// \brief read FLM_BACKEND +/// \return the override, or empty when it is unset +std::string EnvBackend() { + const char* configured = std::getenv("FLM_BACKEND"); + if (!configured || !*configured) return {}; + return configured; +} + +} // namespace + +BackendRegistry& BackendRegistry::instance() { + static BackendRegistry registry; + static std::once_flag once; + // register_builtin_backends must not call instance(), or this deadlocks. + std::call_once(once, [] { register_builtin_backends(registry); }); + return registry; +} + +void BackendRegistry::register_backend(std::string family, std::string id, + BackendFactory factory, + BackendTraits traits) { + if (family.empty()) throw std::runtime_error("backend family is empty"); + if (id.empty()) throw std::runtime_error("backend id is empty"); + if (!factory) throw std::runtime_error("backend factory is null"); + + std::lock_guard lock(mutex_); + auto& per_family = factories_[std::move(family)]; + if (per_family.count(id)) { + throw std::runtime_error("backend '" + id + "' is already registered"); + } + per_family.emplace(std::move(id), Entry{std::move(factory), traits}); +} + +void BackendRegistry::replace_backend(std::string family, std::string id, + BackendFactory factory, + BackendTraits traits) { + if (family.empty()) throw std::runtime_error("backend family is empty"); + if (id.empty()) throw std::runtime_error("backend id is empty"); + if (!factory) throw std::runtime_error("backend factory is null"); + + std::lock_guard lock(mutex_); + factories_[std::move(family)][std::move(id)] = + Entry{std::move(factory), traits}; +} + +std::vector BackendRegistry::available( + const std::string& family) const { + std::lock_guard lock(mutex_); + std::vector ids; + const auto per_family = factories_.find(family); + if (per_family == factories_.end()) return ids; + ids.reserve(per_family->second.size()); + for (const auto& [id, entry] : per_family->second) ids.push_back(id); + return ids; // std::map keeps them sorted +} + +BackendRegistry::Entry BackendRegistry::lookup(const std::string& family, + const std::string& id) const { + { + std::lock_guard lock(mutex_); + const auto per_family = factories_.find(family); + if (per_family != factories_.end()) { + const auto found = per_family->second.find(id); + if (found != per_family->second.end()) return found->second; + } + } + throw std::runtime_error("Model family '" + family + + "' has no backend '" + id + + "'. This build provides: " + + Join(available(family))); +} + +BackendTraits BackendRegistry::traits(const std::string& family, + const std::string& id) const { + return lookup(family, id).traits; +} + +bool BackendRegistry::has(const std::string& family, + const std::string& id) const { + std::lock_guard lock(mutex_); + const auto per_family = factories_.find(family); + return per_family != factories_.end() && per_family->second.count(id) != 0; +} + +std::unique_ptr BackendRegistry::create( + const std::string& family, const std::string& id, + const BackendContext& context) const { + return lookup(family, id).factory(context); +} + +std::string resolve_backend_id(const std::string& family, + const std::string& fallback, + const std::string& requested, + std::string* source) { + std::string chosen; + std::string chosen_source; + if (!requested.empty()) { + chosen = requested; + chosen_source = "--backend"; + } else if (std::string env = EnvBackend(); !env.empty()) { + chosen = std::move(env); + chosen_source = "FLM_BACKEND"; + } else { + chosen = fallback; + chosen_source = "build default"; + } + + if (!BackendRegistry::instance().has(family, chosen)) { + throw std::runtime_error( + "Backend '" + chosen + "' (from " + chosen_source + + ") is not compiled into this build of flm. Model family '" + + family + "' provides: " + + Join(BackendRegistry::instance().available(family))); + } + + if (source) *source = chosen_source; + return chosen; +} + +} // namespace flm::backend diff --git a/src/common/AutoModel/modeling_gemma3.cpp b/src/common/AutoModel/modeling_gemma3.cpp index 86e1d703b..368be6909 100644 --- a/src/common/AutoModel/modeling_gemma3.cpp +++ b/src/common/AutoModel/modeling_gemma3.cpp @@ -10,18 +10,8 @@ /************ Gemma3 family **************/ Gemma3::Gemma3(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Gemma3") {} -void Gemma3::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // model_type == gemma - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); +void Gemma3::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); diff --git a/src/common/AutoModel/modeling_gemma3_text.cpp b/src/common/AutoModel/modeling_gemma3_text.cpp index bb4826be9..f5a4b17a1 100644 --- a/src/common/AutoModel/modeling_gemma3_text.cpp +++ b/src/common/AutoModel/modeling_gemma3_text.cpp @@ -11,19 +11,9 @@ /************ Gemma3_Text_Only family **************/ Gemma3_Text_Only::Gemma3_Text_Only(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Gemma3_Text_Only") {} -void Gemma3_Text_Only::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { +void Gemma3_Text_Only::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // model_type == gemma_text_only - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); diff --git a/src/common/AutoModel/modeling_gemma4_12b.cpp b/src/common/AutoModel/modeling_gemma4_12b.cpp index a6c1213f0..66178cf0a 100644 --- a/src/common/AutoModel/modeling_gemma4_12b.cpp +++ b/src/common/AutoModel/modeling_gemma4_12b.cpp @@ -403,26 +403,19 @@ std::pair parse_gemma4_12b_tool_content(std::string tool_cont /************ Gemma4_12B family **************/ Gemma4_12B::Gemma4_12B(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Gemma4_12B") {} -void Gemma4_12B::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { +void Gemma4_12B::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); - this->q4nx = std::make_unique(this->model_path); - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - // free the q4nx - this->q4nx.reset(); // The soft token budget is a preprocessing constant (processor_config.json // image_seq_length / image_processor.max_soft_tokens), read by the engine // along with the rest of the image front end parameters. { - gemma4_12b_npu* engine = dynamic_cast(this->lm_engine.get()); + gemma4_12b_npu* engine = dynamic_cast(this->lm_engine); if (engine != nullptr && engine->GEMMA4_12B_vision_max_soft_tokens > 0){ this->image_softtoken_budget = (int)engine->GEMMA4_12B_vision_max_soft_tokens; } } - this->lm_engine->clear_context(); this->setup_tokenizer(model_path); this->sampler.reset(); @@ -545,7 +538,7 @@ bool Gemma4_12B::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, } // Process Audios if (message.contains("audios")) { - gemma4_12b_npu *gemma4e_engine = dynamic_cast(this->lm_engine.get()); + gemma4_12b_npu *gemma4e_engine = dynamic_cast(this->lm_engine); for (auto& aud : message["audios"]) { std::string audio_str = aud.get(); audio_data_t audio_data = this->load_audio_base64(audio_str, gemma4e_engine->GEMMA4_12B_audio_sampling_rate, MonoDownmixMode::RMS); @@ -597,7 +590,7 @@ bool Gemma4_12B::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, else { // a pure text, usually from the cli // nlohmann::ordered_json messages; if(input.audios.size() > 0){ - gemma4_12b_npu *gemma4e_engine = dynamic_cast(this->lm_engine.get()); + gemma4_12b_npu *gemma4e_engine = dynamic_cast(this->lm_engine); for (int i = 0; i < input.audios.size(); i++) { std::string audio_str = input.audios[i]; @@ -859,7 +852,7 @@ bool Gemma4_12B::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, // hardware int restore_idx = -1; - gemma4_12b_npu *gemma4_12b_engine = dynamic_cast(this->lm_engine.get()); + gemma4_12b_npu *gemma4_12b_engine = dynamic_cast(this->lm_engine); if (meta_info.restore_allowed) { restore_idx = gemma4_12b_engine->restore(); @@ -1009,7 +1002,7 @@ std::string Gemma4_12B::generate(chat_meta_info_t& meta_info, int length_limit, return result; // if (!this->enable_think) { - // gemma4_12b_npu *gemma4_12b_engine = dynamic_cast(this->lm_engine.get()); + // gemma4_12b_npu *gemma4_12b_engine = dynamic_cast(this->lm_engine); // int checkpoint_idx = gemma4_12b_engine->checkpoint(); // // copy the token history at the checkpoint except the last one token, which is the start // // token for generation and should not be included in the checkpoint history diff --git a/src/common/AutoModel/modeling_gemma4_12b_image.cpp b/src/common/AutoModel/modeling_gemma4_12b_image.cpp index f5c0e8d52..981826730 100644 --- a/src/common/AutoModel/modeling_gemma4_12b_image.cpp +++ b/src/common/AutoModel/modeling_gemma4_12b_image.cpp @@ -283,7 +283,7 @@ void Gemma4_12B::preprocess_image( { //std::cout << "hit preprocess_image, image size: " << image.width << "x" << image.height << ", pixel count: " << (image.width * image.height) << std::endl; - gemma4_12b_npu *lm_engine_gemma4e_ptr = reinterpret_cast(this->lm_engine.get()); + gemma4_12b_npu *lm_engine_gemma4e_ptr = reinterpret_cast(this->lm_engine); int max_patches = this->image_softtoken_budget * lm_engine_gemma4e_ptr->GEMMA4_12B_vision_pooling_kernel_size * lm_engine_gemma4e_ptr->GEMMA4_12B_vision_pooling_kernel_size; // first, do_resize diff --git a/src/common/AutoModel/modeling_gemma4e.cpp b/src/common/AutoModel/modeling_gemma4e.cpp index 99ae20672..710697483 100644 --- a/src/common/AutoModel/modeling_gemma4e.cpp +++ b/src/common/AutoModel/modeling_gemma4e.cpp @@ -412,10 +412,6 @@ std::pair parse_gemma4e_tool_content(std::string tool_content /************ Gemma4e family **************/ Gemma4e::Gemma4e(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Gemma4e") {} -void Gemma4e::create_engine() { - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); -} - /// Reads the shared set of engine constants off a concrete engine type. /// gemma4e_npu and gemma4e_flash are unrelated types that happen to expose the /// same member names, so this is templated rather than taking a common base. @@ -437,15 +433,11 @@ static gemma4e_engine_config_t read_gemma4e_engine_config(causal_lm* engine) { } gemma4e_engine_config_t Gemma4e::engine_config() const { - return read_gemma4e_engine_config(this->lm_engine.get()); + return read_gemma4e_engine_config(this->lm_engine); } gemma4e_engine_config_t Gemma4e_Flash::engine_config() const { - return read_gemma4e_engine_config(this->lm_engine.get()); -} - -void Gemma4e_Flash::create_engine() { - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); + return read_gemma4e_engine_config(this->lm_engine); } int Gemma4e_Flash::_pin_system_prefix(const std::string& system_text) { @@ -648,17 +640,9 @@ std::string Gemma4e_Flash::generate_with_prompt(chat_meta_info_t& meta_info, lm_ return this->generate(meta_info, length_limit, os); } -void Gemma4e::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { +void Gemma4e::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - this->create_engine(); - - this->lm_engine->load_weights(*this->q4nx); - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); diff --git a/src/common/AutoModel/modeling_gpt_oss.cpp b/src/common/AutoModel/modeling_gpt_oss.cpp index 51228bdbe..31ad039f9 100644 --- a/src/common/AutoModel/modeling_gpt_oss.cpp +++ b/src/common/AutoModel/modeling_gpt_oss.cpp @@ -9,13 +9,9 @@ GPT_OSS::GPT_OSS(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "gpt-oss") {} -void GPT_OSS::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { +void GPT_OSS::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { this->model_path = model_path; - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - this->q4nx = std::make_unique(this->model_path); - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - this->lm_engine->load_weights(*this->q4nx); - this->q4nx.reset(); + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->tokenizer = std::make_unique(model_path); this->setup_tokenizer(model_path); @@ -80,7 +76,7 @@ bool GPT_OSS::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std // hardware int restore_idx = -1; - gpt_oss_npu *gpt_oss_engine = dynamic_cast(this->lm_engine.get()); + gpt_oss_npu *gpt_oss_engine = dynamic_cast(this->lm_engine); if (meta_info.restore_allowed) { restore_idx = gpt_oss_engine->restore(); this->total_tokens = restore_idx; diff --git a/src/common/AutoModel/modeling_hunyuan.cpp b/src/common/AutoModel/modeling_hunyuan.cpp index 5d7795cff..2289b7060 100644 --- a/src/common/AutoModel/modeling_hunyuan.cpp +++ b/src/common/AutoModel/modeling_hunyuan.cpp @@ -19,18 +19,8 @@ Hunyuan::Hunyuan(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, " this->forward_on_eos = false; } -void Hunyuan::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - - // free the mmap'd weights immediately - this->q4nx.reset(); - - this->lm_engine->clear_context(); +void Hunyuan::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); @@ -158,7 +148,7 @@ bool Hunyuan::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std // rewind to the pinned system turn instead of clearing: _shared_insert // then prefix-matches against the history and prefills only the tail. // The token history has to move back with the cache or that match fails. - hunyuan_npu* engine = dynamic_cast(this->lm_engine.get()); + hunyuan_npu* engine = dynamic_cast(this->lm_engine); this->total_tokens = engine->restore(); this->token_history = this->system_his; this->checkpoint_his = this->system_his; @@ -231,7 +221,7 @@ int Hunyuan::_pin_system_prefix(const std::string& system_text) { // prompt is prefix-matched against this->system_his = this->token_history; this->checkpoint_his = this->token_history; - hunyuan_npu* engine = dynamic_cast(this->lm_engine.get()); + hunyuan_npu* engine = dynamic_cast(this->lm_engine); engine->checkpoint(); this->system_tokens = static_cast(shared); diff --git a/src/common/AutoModel/modeling_lfm2.cpp b/src/common/AutoModel/modeling_lfm2.cpp index 82384c7a5..349eaf541 100644 --- a/src/common/AutoModel/modeling_lfm2.cpp +++ b/src/common/AutoModel/modeling_lfm2.cpp @@ -12,18 +12,8 @@ /************ LFM2 family **************/ LFM2::LFM2(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "LFM2") {} -void LFM2::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // model_type == llama - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); +void LFM2::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); @@ -267,18 +257,8 @@ StreamResult LFM2::parse_stream_content(const std::string content) { /*********** LFM2_5_TK family ***********/ LFM2_5_TK::LFM2_5_TK(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "LFM2_5_TK") {} -void LFM2_5_TK::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // model_type == llama - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); +void LFM2_5_TK::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); @@ -337,7 +317,7 @@ bool LFM2_5_TK::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, s // hardware int restore_idx = -1; - lfm2_npu *lfm2_engine = dynamic_cast(this->lm_engine.get()); + lfm2_npu *lfm2_engine = dynamic_cast(this->lm_engine); if (meta_info.restore_allowed) { restore_idx = lfm2_engine->restore(); diff --git a/src/common/AutoModel/modeling_llama3.cpp b/src/common/AutoModel/modeling_llama3.cpp index 5ebcbd1dd..e9656f565 100644 --- a/src/common/AutoModel/modeling_llama3.cpp +++ b/src/common/AutoModel/modeling_llama3.cpp @@ -10,19 +10,8 @@ /************ Llama3 family **************/ Llama3::Llama3(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Llama3") {} -void Llama3::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // model_type == llama - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - - this->lm_engine->clear_context(); +void Llama3::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); @@ -92,18 +81,8 @@ std::string Llama3::generate_with_prompt(chat_meta_info_t& meta_info, lm_uniform /************ DeepSeek_r1_8b family **************/ DeepSeek_r1_8b::DeepSeek_r1_8b(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst) {} -void DeepSeek_r1_8b::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // model_type == llama - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); +void DeepSeek_r1_8b::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); @@ -157,7 +136,7 @@ bool DeepSeek_r1_8b::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& inp this->profiler_list[TKOEN_ENCODE_TIME].stop(tokens.size()); // hardware int restore_idx = -1; - llama_npu *llama_engine = dynamic_cast(this->lm_engine.get()); + llama_npu *llama_engine = dynamic_cast(this->lm_engine); if (meta_info.restore_allowed) { restore_idx = llama_engine->restore(); this->total_tokens = restore_idx; diff --git a/src/common/AutoModel/modeling_nanbeige.cpp b/src/common/AutoModel/modeling_nanbeige.cpp index 8a8532211..7eeac2a07 100644 --- a/src/common/AutoModel/modeling_nanbeige.cpp +++ b/src/common/AutoModel/modeling_nanbeige.cpp @@ -10,19 +10,8 @@ /************ Nanbeige family **************/ Nanbeige::Nanbeige(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Nanbeige") {} -void Nanbeige::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // model_type == nanbeige - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - - this->lm_engine->clear_context(); +void Nanbeige::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); @@ -88,7 +77,7 @@ bool Nanbeige::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, st // hardware int restore_idx = -1; - nanbeige_npu *nanbeige_engine = dynamic_cast(this->lm_engine.get()); + nanbeige_npu *nanbeige_engine = dynamic_cast(this->lm_engine); if (meta_info.restore_allowed) { restore_idx = nanbeige_engine->restore(); diff --git a/src/common/AutoModel/modeling_phi4.cpp b/src/common/AutoModel/modeling_phi4.cpp index 06c164f51..eb8849cdd 100644 --- a/src/common/AutoModel/modeling_phi4.cpp +++ b/src/common/AutoModel/modeling_phi4.cpp @@ -1,125 +1,160 @@ -/// \file phi4.cpp -/// \brief phi4 class -/// \author FastFlowLM Team -/// \date 2025-09-04 -/// \version 0.9.25 -/// \note This is a source file for the phi4 class - +/// \file modeling_phi4.cpp +/// \brief Phi-4 frontend +/// \note Every rule that used to be branched on here -- the rai decode cap, +/// poisoning, the no-preemption rule, the package-verified stop ids, the +/// separate decode loop -- now lives behind flm::backend::ModelBackend. #include "AutoModel/modeling_phi4.hpp" +#include "utils/file_access.hpp" + +#include +#include +#include +#include + +namespace { + +/// \brief read a JSON file, recording the open for the file-access audit +/// \param path the file to read +/// \return the parsed document +nlohmann::json ReadJson(const std::filesystem::path& path) { + flm::file_access::ObserveOpen(path); + std::ifstream input(path, std::ios::binary); + if (!input) throw std::runtime_error("Cannot open " + path.string()); + try { + return nlohmann::json::parse(input); + } catch (const std::exception& error) { + throw std::runtime_error("Cannot parse " + path.string() + ": " + error.what()); + } +} -/************ Phi4 family **************/ -Phi4::Phi4(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Phi4") {} - -void Phi4::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // model_type == phi4 - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - - this->lm_engine->clear_context(); - this->setup_tokenizer(model_path); - this->sampler.reset(); - +void ConfigureSampler(Phi4& model) { sampler_config config; config.top_k = 40; config.top_p = 0.9; config.min_p = 0.1; config.temperature = 0.8; + model.set_sampler(config); +} +} // namespace - this->set_sampler(config); - for (size_t i = 0; i < PROFILER_TYPE_NUM; i++) { - this->profiler_list[i].reset(); - } +Phi4::Phi4(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Phi4") {} + +void Phi4::load_model(std::string model_path, json model_info, + int default_context_length, bool enable_preemption, + const std::string& backend) { + // Read once here and hand the parse to both consumers. The backend needs it + // to cross-validate its package and the tokenizer setup below needs it for + // the chat template, but the model directory layout is this frontend's + // knowledge -- a backend that opens the directory itself duplicates both + // the read and that knowledge. + const nlohmann::json tokenizer_config = + ReadJson(std::filesystem::path(model_path) / "tokenizer_config.json"); + this->_shared_load_backend(model_path, model_info, default_context_length, + enable_preemption, backend, &tokenizer_config); + this->setup_tokenizer(tokenizer_config); + this->sampler.reset(); + ConfigureSampler(*this); + for (auto& item : profiler_list) item.reset(); } -void Phi4::setup_tokenizer(std::string model_path) { - // load tokenizer configurations - #ifdef _WIN32 - std::string tokenizer_config_path = model_path + "\\tokenizer_config.json"; - #else - std::string tokenizer_config_path = model_path + "/tokenizer_config.json"; - #endif - std::ifstream fs_config(tokenizer_config_path, std::ios::in | std::ios::binary); - if (fs_config.fail()) { - std::cerr << "Cannot open " << tokenizer_config_path << std::endl; - exit(1); - } - std::string data_config; - fs_config.seekg(0, std::ios::end); - size_t size_config = static_cast(fs_config.tellg()); - fs_config.seekg(0, std::ios::beg); - data_config.resize(size_config); - fs_config.read(data_config.data(), size_config); - fs_config.close(); - auto tokenizer_config = nlohmann::json::parse(data_config); - this->has_bos_token = false; - // load chat template - this->chat_tmpl = std::make_unique( - tokenizer_config["chat_template"], - "", - "" - ); - - if (this->has_bos_token) { - this->bos_token_id = tokenizer_config["bos_token_id"].get(); - } - else { - this->bos_token_id = -1; +void Phi4::setup_tokenizer(const nlohmann::json& config) { + if (!config.contains("chat_template") || !config["chat_template"].is_string()) + throw std::invalid_argument("Phi-4 tokenizer_config.json requires a string chat_template"); + + // Preserve the legacy Phi-4 contract: minja receives no textual BOS/EOS, + // and there is no automatic BOS token. + auto chat = std::make_unique( + config["chat_template"].get(), "", ""); + + std::vector eos; + // A backend that cross-validated its own package proved its stop ids from + // independent sources, so those win over tokenizer_config.json. + const auto forced = this->backend_ ? this->backend_->forced_eos_ids() + : std::nullopt; + if (forced) { + eos = *forced; + } else { + if (!config.contains("eos_token_id")) + throw std::invalid_argument("Phi-4 tokenizer_config.json requires eos_token_id"); + const auto& ids = config["eos_token_id"]; + if (ids.is_number_integer()) eos.push_back(ids.get()); + else if (ids.is_array()) for (const auto& id : ids) eos.push_back(id.get()); + else throw std::invalid_argument("Phi-4 tokenizer_config.json eos_token_id must be integer or array"); } - this->eos_token = ""; - for (auto& token : tokenizer_config["eos_token_id"]) { - this->eos_token_ids.push_back(token.get()); - } - this->user_system_prompt = ""; - this->extra_context["user_system_prompt"] = this->user_system_prompt; + + has_bos_token = false; + bos_token_id = -1; + eos_token.clear(); + eos_token_ids = std::move(eos); + chat_tmpl = std::move(chat); + user_system_prompt.clear(); + extra_context["user_system_prompt"] = user_system_prompt; } -std::string Phi4::apply_chat_template(nlohmann::ordered_json& messages, nlohmann::ordered_json tools) { +std::string Phi4::apply_chat_template(nlohmann::ordered_json& messages, + nlohmann::ordered_json) { minja::chat_template_inputs inputs; inputs.add_generation_prompt = true; inputs.messages = messages; - inputs.extra_context = this->extra_context; - return this->chat_tmpl->apply(inputs); + inputs.extra_context = extra_context; + return chat_tmpl->apply(inputs); +} + +void Phi4::fail_inference() { + const bool poisoned = backend_ && backend_->poisoned(); + _shared_after_inference_failure(poisoned); + throw ModelRequestError(500, true, poisoned + ? "Inference failed; unload/reload is required because the model is poisoned" + : "Inference failed; the current conversation was cleared"); } -bool Phi4::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled) { - // preprocess - this->profiler_list[TKOEN_ENCODE_TIME].start(); - std::string templated_text; - if (input.messages.empty() && input.prompt.empty()) { - header_print("WARNING", "No messages or prompt provided"); +bool Phi4::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, + std::function is_cancelled) { + _shared_guard_poisoned(); + + profiler_list[TKOEN_ENCODE_TIME].start(); + std::string rendered; + if (input.messages.empty() && input.prompt.empty()) return false; + if (!input.messages.empty()) rendered = apply_chat_template(input.messages); + else { + nlohmann::ordered_json messages = nlohmann::ordered_json::array(); + messages.push_back({{"role", "user"}, {"content", input.prompt}}); + rendered = apply_chat_template(messages); + } + std::vector tokens = tokenizer->encode(rendered); + profiler_list[TKOEN_ENCODE_TIME].stop(tokens.size()); + + if (is_cancelled()) { + meta_info.stop_reason = CANCEL_DETECTED; return false; } - if (!input.messages.empty()) { // already a formated messages, usually from REST API - templated_text = this->apply_chat_template(input.messages); + try { + return _shared_insert(meta_info, tokens, std::move(is_cancelled), nullptr, + 0, input.requested_max_new_tokens); + } catch (const ModelRequestError&) { + throw; + } catch (...) { + fail_inference(); } - else if (!input.prompt.empty()) { // a pure text, usually from the cli - nlohmann::ordered_json messages; +} - messages.push_back({ {"role", "user"}, {"content", input.prompt} }); - templated_text = this->apply_chat_template(messages); +std::string Phi4::generate(chat_meta_info_t& meta_info, int length_limit, + std::ostream& os, + std::function is_cancelled) { + _shared_guard_poisoned(); + try { + return _shared_generate(meta_info, length_limit, os, std::move(is_cancelled)); + } catch (const ModelRequestError&) { + throw; + } catch (...) { + fail_inference(); } - - std::vector tokens = this->tokenizer->encode(templated_text); - this->profiler_list[TKOEN_ENCODE_TIME].stop(tokens.size()); - // hardware - - return this->_shared_insert(meta_info, tokens, is_cancelled); } - -std::string Phi4::generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled) { - return this->_shared_generate(meta_info, length_limit, os, is_cancelled); +std::string Phi4::generate_with_prompt(chat_meta_info_t& meta_info, + lm_uniform_input_t& input, + int length_limit, + std::ostream& os) { + if (!insert(meta_info, input)) return {}; + return generate(meta_info, length_limit, os); } - -std::string Phi4::generate_with_prompt(chat_meta_info_t& meta_info, lm_uniform_input_t& input, int length_limit, std::ostream& os) { - if (!this->insert(meta_info, input)) { - return ""; - } - return this->_shared_generate(meta_info, length_limit, os); -} \ No newline at end of file diff --git a/src/common/AutoModel/modeling_qwen2.cpp b/src/common/AutoModel/modeling_qwen2.cpp index 11b5fd513..f5a4bf922 100644 --- a/src/common/AutoModel/modeling_qwen2.cpp +++ b/src/common/AutoModel/modeling_qwen2.cpp @@ -11,19 +11,8 @@ /************ Qwen2 family **************/ Qwen2::Qwen2(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst) {} -void Qwen2::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - - // lm_config->get("model_type", "") == qwen2 - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); +void Qwen2::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); diff --git a/src/common/AutoModel/modeling_qwen2vl.cpp b/src/common/AutoModel/modeling_qwen2vl.cpp index 4ca700d9b..1b8d75a49 100644 --- a/src/common/AutoModel/modeling_qwen2vl.cpp +++ b/src/common/AutoModel/modeling_qwen2vl.cpp @@ -14,17 +14,8 @@ /************ Qwen2VL family **************/ Qwen2VL::Qwen2VL(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Qwen2VL") {} -void Qwen2VL::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // lm_config->get("model_type", "") == qwen2 - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); +void Qwen2VL::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); diff --git a/src/common/AutoModel/modeling_qwen3.cpp b/src/common/AutoModel/modeling_qwen3.cpp index 39a29f8ac..dc083bfd2 100644 --- a/src/common/AutoModel/modeling_qwen3.cpp +++ b/src/common/AutoModel/modeling_qwen3.cpp @@ -12,18 +12,8 @@ /************ Qwen3 family **************/ Qwen3::Qwen3(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Qwen3") {} -void Qwen3::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // lm_config->get("model_type", "") == qwen3 - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); +void Qwen3::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); @@ -81,7 +71,7 @@ bool Qwen3::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std:: // hardware int restore_idx = -1; - qwen3_npu *qwen3_engine = dynamic_cast(this->lm_engine.get()); + qwen3_npu *qwen3_engine = dynamic_cast(this->lm_engine); if (meta_info.restore_allowed) { restore_idx = qwen3_engine->restore(); @@ -309,19 +299,8 @@ StreamResult Qwen3::parse_stream_content(const std::string content) { /************ Qwen3_IT family **************/ Qwen3_IT::Qwen3_IT(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst) {} -void Qwen3_IT::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - - // lm_config->get("model_type", "") == qwen3 - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); +void Qwen3_IT::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); @@ -459,19 +438,8 @@ StreamResult Qwen3_IT::parse_stream_content(const std::string content) { /************ Qwen3_TK family **************/ Qwen3_TK::Qwen3_TK(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst) {} -void Qwen3_TK::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - - // lm_config->get("model_type", "") == qwen3 - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); +void Qwen3_TK::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); @@ -524,7 +492,7 @@ bool Qwen3_TK::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, st // hardware int restore_idx = -1; - qwen3_npu *qwen3_engine = dynamic_cast(this->lm_engine.get()); + qwen3_npu *qwen3_engine = dynamic_cast(this->lm_engine); if (meta_info.restore_allowed) { restore_idx = qwen3_engine->restore(); @@ -681,18 +649,8 @@ StreamResult Qwen3_TK::parse_stream_content(const std::string content) { /************ DeepSeek_r1_0528_8b family **************/ DeepSeek_r1_0528_8b::DeepSeek_r1_0528_8b(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst) {} -void DeepSeek_r1_0528_8b::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // model_type == llama - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); +void DeepSeek_r1_0528_8b::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); @@ -746,7 +704,7 @@ bool DeepSeek_r1_0528_8b::insert(chat_meta_info_t& meta_info, lm_uniform_input_t // hardware int restore_idx = -1; - qwen3_npu *qwen3_engine = dynamic_cast(this->lm_engine.get()); + qwen3_npu *qwen3_engine = dynamic_cast(this->lm_engine); if (meta_info.restore_allowed) { restore_idx = qwen3_engine->restore(); this->total_tokens = restore_idx; diff --git a/src/common/AutoModel/modeling_qwen3_5_omni.cpp b/src/common/AutoModel/modeling_qwen3_5_omni.cpp index 28695786d..2555942d0 100644 --- a/src/common/AutoModel/modeling_qwen3_5_omni.cpp +++ b/src/common/AutoModel/modeling_qwen3_5_omni.cpp @@ -63,7 +63,7 @@ void Qwen3_5_Omni::setup_tokenizer(std::string model_path) { this->extra_context["user_system_prompt"] = this->user_system_prompt; } -void Qwen3_5_Omni::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { +void Qwen3_5_Omni::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { if (this->is_model_loaded && this->model_path == model_path) { header_print("FLM", "Model already loaded: " << this->model_path); return; diff --git a/src/common/AutoModel/modeling_qwen3_5vl.cpp b/src/common/AutoModel/modeling_qwen3_5vl.cpp index 21b759ef5..c7a692e4c 100644 --- a/src/common/AutoModel/modeling_qwen3_5vl.cpp +++ b/src/common/AutoModel/modeling_qwen3_5vl.cpp @@ -13,17 +13,8 @@ /************ Qwen3_5VL family **************/ Qwen3_5VL::Qwen3_5VL(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Qwen3_5VL") {} -void Qwen3_5VL::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // lm_config->get("model_type", "") == qwen3 - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); +void Qwen3_5VL::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); @@ -229,7 +220,7 @@ bool Qwen3_5VL::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, s if (prefix_skip_count > 0 && !image_payload.images.empty()) { // Per-image bf16 footprint depends on runtime patch/temporal // config carried by the engine. - auto* eng = reinterpret_cast(this->lm_engine.get()); + auto* eng = reinterpret_cast(this->lm_engine); const unsigned patch_size = eng->QWEN3_5_PATCH_SIZE; const unsigned temporal_patch = eng->QWEN3_5_TEMPORAL_PATCH_SIZE; @@ -288,7 +279,7 @@ bool Qwen3_5VL::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, s // hardware int restore_idx = -1; - qwen3_5vl_npu *qwen3_5vl_engine = dynamic_cast(this->lm_engine.get()); + qwen3_5vl_npu *qwen3_5vl_engine = dynamic_cast(this->lm_engine); const bool has_images = image_payload.num_images > 0; if (meta_info.restore_allowed) { @@ -446,7 +437,7 @@ std::string Qwen3_5VL::generate_with_prompt(chat_meta_info_t& meta_info, lm_unif return ""; } header_print("FLM", "Prompt inserted, starting generation..."); - qwen3_5vl_npu* qwen35_engine = dynamic_cast(this->lm_engine.get()); + qwen3_5vl_npu* qwen35_engine = dynamic_cast(this->lm_engine); int checkpoint_idx = qwen35_engine->checkpoint(); int restore_idx = qwen35_engine->restore(); header_print_r("FLM", "Checkpoint before generation: " << checkpoint_idx << ", restore point: " << restore_idx << ", user context length: " << this->token_history.size()); diff --git a/src/common/AutoModel/modeling_qwen3_5vl_image.cpp b/src/common/AutoModel/modeling_qwen3_5vl_image.cpp index 1ee607fb1..19cb67871 100644 --- a/src/common/AutoModel/modeling_qwen3_5vl_image.cpp +++ b/src/common/AutoModel/modeling_qwen3_5vl_image.cpp @@ -191,7 +191,7 @@ void Qwen3_5VL::preprocess_image(qwen3_5vl_image_t& image, std::vector &pi int resized_width; // do the automatically resizing in here - qwen3_5vl_npu* lm_engine_qwen3_5_ptr = reinterpret_cast(this->lm_engine.get()); + qwen3_5vl_npu* lm_engine_qwen3_5_ptr = reinterpret_cast(this->lm_engine); smart_resize( height, width, resized_height, resized_width, diff --git a/src/common/AutoModel/modeling_qwen3_6_moe.cpp b/src/common/AutoModel/modeling_qwen3_6_moe.cpp index 5fc8ead51..4c00fe310 100644 --- a/src/common/AutoModel/modeling_qwen3_6_moe.cpp +++ b/src/common/AutoModel/modeling_qwen3_6_moe.cpp @@ -13,17 +13,8 @@ /************ Qwen3_6_MOE family **************/ Qwen3_6_MOE::Qwen3_6_MOE(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Qwen3_6_MOE") {} -void Qwen3_6_MOE::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // lm_config->get("model_type", "") == qwen3 - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); - - this->lm_engine->load_weights(*this->q4nx); - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); +void Qwen3_6_MOE::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); @@ -229,7 +220,7 @@ bool Qwen3_6_MOE::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, if (prefix_skip_count > 0 && !image_payload.images.empty()) { // Per-image bf16 footprint depends on runtime patch/temporal // config carried by the engine. - auto* eng = reinterpret_cast(this->lm_engine.get()); + auto* eng = reinterpret_cast(this->lm_engine); const unsigned patch_size = eng->QWEN3_6_MOE_PATCH_SIZE; const unsigned temporal_patch = eng->QWEN3_6_MOE_TEMPORAL_PATCH_SIZE; @@ -288,7 +279,7 @@ bool Qwen3_6_MOE::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, // hardware int restore_idx = -1; - qwen3_6_moe_npu *qwen3_6_moe_engine = dynamic_cast(this->lm_engine.get()); + qwen3_6_moe_npu *qwen3_6_moe_engine = dynamic_cast(this->lm_engine); const bool has_images = image_payload.num_images > 0; if (meta_info.restore_allowed) { @@ -446,7 +437,7 @@ std::string Qwen3_6_MOE::generate_with_prompt(chat_meta_info_t& meta_info, lm_un return ""; } header_print("FLM", "Prompt inserted, starting generation..."); - qwen3_6_moe_npu* qwen36_engine = dynamic_cast(this->lm_engine.get()); + qwen3_6_moe_npu* qwen36_engine = dynamic_cast(this->lm_engine); int checkpoint_idx = qwen36_engine->checkpoint(); int restore_idx = qwen36_engine->restore(); header_print_r("FLM", "Checkpoint before generation: " << checkpoint_idx << ", restore point: " << restore_idx << ", user context length: " << this->token_history.size()); diff --git a/src/common/AutoModel/modeling_qwen3_6_moe_image.cpp b/src/common/AutoModel/modeling_qwen3_6_moe_image.cpp index 2f0150a7c..e40472bae 100644 --- a/src/common/AutoModel/modeling_qwen3_6_moe_image.cpp +++ b/src/common/AutoModel/modeling_qwen3_6_moe_image.cpp @@ -213,7 +213,7 @@ void Qwen3_6_MOE::preprocess_image(qwen3_6_moe_image_t& image, std::vector return; } - qwen3_6_moe_npu* lm_engine_qwen3_6_ptr = reinterpret_cast(this->lm_engine.get()); + qwen3_6_moe_npu* lm_engine_qwen3_6_ptr = reinterpret_cast(this->lm_engine); smart_resize( height, width, resized_height, resized_width, diff --git a/src/common/AutoModel/modeling_qwen3vl.cpp b/src/common/AutoModel/modeling_qwen3vl.cpp index bd69d6e67..de2ee1d0e 100644 --- a/src/common/AutoModel/modeling_qwen3vl.cpp +++ b/src/common/AutoModel/modeling_qwen3vl.cpp @@ -13,25 +13,8 @@ /************ Qwen3VL family **************/ Qwen3VL::Qwen3VL(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Qwen3VL") {} -void Qwen3VL::create_engine() { - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); -} - -void Qwen3VL_Flash::create_engine() { - this->lm_engine = std::make_unique(*this->lm_config, this->npu.get(), this->MAX_L); -} - -void Qwen3VL::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption) { - this->_shared_load_model(model_path, model_info, default_context_length, enable_preemption); - - this->q4nx = std::make_unique(this->model_path); - // lm_config->get("model_type", "") == qwen3 - this->create_engine(); - - this->lm_engine->load_weights(*this->q4nx); - //free the q4nx - this->q4nx.reset(); - this->lm_engine->clear_context(); +void Qwen3VL::load_model(std::string model_path, json model_info, int default_context_length, bool enable_preemption, const std::string& backend) { + this->_shared_load_backend(model_path, model_info, default_context_length, enable_preemption, backend); this->setup_tokenizer(model_path); this->sampler.reset(); diff --git a/src/common/models/README.md b/src/common/models/README.md new file mode 100644 index 000000000..eed016c71 --- /dev/null +++ b/src/common/models/README.md @@ -0,0 +1,347 @@ +# Adding a model on the rai backend + +How to bring up a new model family on **rai**, the backend that reaches its +kernels through ryzenai-corelib, using Phi-4 as the worked example. Phi-4 is the +only model on this path today, so every file named below has a `phi4` counterpart +you can read straight through. + +A backend names *where the kernels come from*: `flm` is FastFlowLM's own kernel +flow, `rai` is corelib. That is a separate axis from which silicon a build +targets, which is `utils::npu_platform` (`stx`, `aie_next`). The two line up +one-to-one today — a rai build is an aie_next build — but they are different +questions and they get different names, so that the day they stop lining up is +not the day every string in the tree starts lying. + +This is a contributor document. For *using* a backend once it exists — `--backend`, +`FLM_BACKEND`, precedence — see [`docs/docs/instructions/cli.md`](../../../docs/docs/instructions/cli.md). + +--- + +## 0. Before you start + +| | | +|---|---| +| **Platform** | Windows only. `FLM_ENABLE_RAI` is rejected at configure time elsewhere ([`CMakeLists.txt:54`](../../CMakeLists.txt#L54)). | +| **Hardware** | An aie_next NPU. There is no simulator; a wrong shape shows up as garbage output, not an error. | +| **corelib headers** | Exactly **0.5.0**. [`corelib_api.hpp`](../../include/rai/corelib_api.hpp) `#error`s on any other version — deliberately, because the C ABI has changed shape between patch releases. | +| **Weights** | A GGUF the vendor kernels can requantize. Phi-4 uses Q8_0; the corelib entry points are `*_create_gguf_requantized`. | + +Configure with: + +```powershell +cmake -B build -S src -DFLM_ENABLE_RAI=ON +# RYZENAI_CORELIB_INCLUDE_DIR / RYZENAI_CORELIB_LIB_DIR are found automatically +# when they are on the default paths; otherwise pass them. +``` + +That builds the `flm_rai` static library and links it into `flm`. The +library carries `FLM_ENABLE_RAI=1` as a **PUBLIC** compile definition, so +everything that links it sees the `#if` guards flip. + +--- + +## 1. Pick the family + +You need one name, not two. + +- **family** — `details.family` in [`model_list.json`](../../model_list.json), e.g. `"phi4"`. + It selects the *frontend* (the chat template, the tokenizer contract, the sampler). +- **backend id** — already decided: `rai`. It names the kernel provider, and the + constant is [`flm::backend::kRaiBackendId`](../../include/AutoModel/model_backend.hpp). + You do not mint one. + +So a corelib engine for a family that already exists (Phi-4 today) needs no new +name at all — it registers the family's `rai` backend. A genuinely new +architecture needs a new family. + +A family has at most one backend per provider. If you find yourself wanting two +engines behind the same provider, the choice belongs in the catalog as two +entries, not in the backend id. + +--- + +## 2. Lay out the files + +``` +src/include/models//rai/ headers for the corelib implementation +src/common/models//rai/ the corelib implementation +``` + +The folders are named after the kernel provider, not the silicon. FastFlowLM's +own engines have no folder here — they ship as prebuilt libraries under +`lib//` and their headers stay at `models//`. Headers mirror the +source split, so an `#include` says which provider it belongs to +(`models/phi4/rai/phi4_rai_gguf.hpp` against `models/phi4/phi4_npu.hpp`), and a +grep for `models//rai/` finds everything the rai path pulls in. + +**Do not edit any `CMakeLists.txt` for this.** [`models_sources.cmake`](models_sources.cmake) +globs `*/rai/*.cpp`, and that glob is what `flm_rai` compiles. Creating the +folder is the whole registration step. + +Phi-4's rai side is five translation units, and the split is worth copying: + +| file | what belongs in it | +|---|---| +| `_rai_gguf.cpp` | Open and validate the GGUF. Tensor lookup by name, shape checks, metadata, and the cross-validation against `config.json` / `tokenizer.json` / `tokenizer_config.json`. **No device code.** | +| `_rai_shape_plan.cpp` | Ask corelib how it wants each operator padded (`matmul_pad_shape`, `ssmlp_pad_rows`, …) and cache one row-extent record per live row count. Built once per engine. | +| `_rai_host.cpp` | The math corelib does not do: Q8 embedding-row decode, RMS norm, f32→bf16, RoPE tables. Plain CPU, unit-testable, no corelib types in the signatures. | +| `_rai.cpp` | The `causal_lm` subclass. Owns the device tensors, the stream and the decode loop. | +| `_rai_backend.cpp` | The `ModelBackend`. Construction order and execution policy. | + +Keeping the GGUF and host layers free of corelib types is what lets you test them +without hardware. + +--- + +## 3. The engine: a `causal_lm` subclass + +Model it on [`phi4_rai.hpp`](../../include/models/phi4/rai/phi4_rai.hpp). +Two rules matter more than the rest. + +### `causal_lm.hpp` is a frozen ABI + +`src/lib/xrt` and `src/lib/hrx` ship ~20 **prebuilt** engine libraries compiled +against today's [`causal_lm.hpp`](../../include/causal_lm.hpp). Their vtables are +emitted inside those binaries. Adding, removing or reordering a virtual there +shifts vtable slots and corrupts dispatch **at runtime, with no compiler error**. + +So: you implement `causal_lm` as it stands. You do not change it. The same +applies to `buffer.hpp`, `tensor_2d.hpp`, `lm_config.hpp`, `q4_npu_eXpress.hpp` +and `npu_utils/*`. If a change to any of those looks necessary, the seam you +actually want is `ModelBackend` (§4), which sits above `causal_lm` and is compiled +from source. + +### `load_weights(Q4NX&)` is a shim + +It is pure virtual in the frozen header, but it describes FastFlowLM's own weight +format, which a GGUF engine does not have. Implement it as a throwing stub with a +comment saying why: + +```cpp +// An ABI shim, not a capability. load_weights is pure virtual in causal_lm.hpp, +// which is frozen because the engine libraries in src/lib/ are prebuilt +// against it. Nothing calls this: FlmBackend loads weights through the +// concrete engine type, and this engine's weights come from the GGUF package it +// was constructed with. See AutoModel/model_backend.hpp. +void _rai::load_weights(Q4NX&) { throw std::runtime_error(...); } +``` + +Nothing calls it: `FlmBackend` calls `load_weights` through the *concrete* +engine type, never through a `causal_lm*`. + +### Everything else + +- Take the GGUF package and the `CorelibRuntime` by `shared_ptr` in the constructor + and hold both for the engine's lifetime. corelib objects must not outlive the + API they were created from. +- Create weights concurrently. Requantizing the 161 weights is effectively the + whole of model load, and the creates are independent — each reads its own + mapped range and produces its own object — so they run across a pool + (`kWeightCreateConcurrency`). The per-create thread hint + (`kRequantizeThreads`) stays at corelib's default of one so the two forms of + parallelism do not multiply into an oversubscribed machine. See + [`phi4_rai_constants.hpp`](../../include/models/phi4/rai/phi4_rai_constants.hpp). +- Expose `bool poisoned() const noexcept`. A corelib failure mid-decode usually + leaves device state that only a reload can clear; the backend surfaces this and + `AutoModel` turns it into a 500 that asks for an unload/reload. +- Put every magic number in a `_rai_constants.hpp` with a comment on + where it came from. + +--- + +## 4. The backend: policy, not just construction + +[`ModelBackend`](../../include/AutoModel/model_backend.hpp) owns one engine **and +every rule for driving it**. Its defaults describe the FastFlowLM NPU engines, so +you override only what differs. Phi-4's corelib backend +([`phi4_rai_backend.cpp`](phi4/rai/phi4_rai_backend.cpp)) overrides six: + +| override | corelib value | why | +|---|---|---| +| `id()` | `"rai"` | the kernel provider; what `--backend` matches | +| `detail()` | `runtime_->loaded_library_path()` | provenance line in `flm show` | +| `max_decode_length()` | `4095` | corelib's own decode window | +| `supports_preemption()` | `false` | no checkpoint/restore on this path | +| `forwards_past_eos()` | `false` | the extra post-EOS `forward()` the FLM engines want would exceed the window | +| `forced_eos_ids()` | `{200020, 199999}` | proven by three sources agreeing in `ValidatePhi4Contract`, which beats `tokenizer_config.json` alone | + +`poisoned()` forwards to the engine. + +### Constructor order + +Validate everything *before* you touch the device, so a mismatched package fails +while nothing has been allocated: + +1. reject preemption / an out-of-range context length, +2. read `config.json`, `tokenizer.json`, `tokenizer_config.json`, +3. `Open` the GGUF and cross-validate it against all three, +4. `CorelibRuntime::GetOrCreate(...)`, +5. construct the engine, `clear_context()`. + +Declare the runtime `shared_ptr` **before** the engine member so it is destroyed +after it, and reset the engine explicitly in the destructor. + +### Traits: what the frontend must know *before* the backend exists + +`BackendTraits` cannot be a virtual on `ModelBackend` — the frontend consults it +while assembling the `BackendContext`. Keep it `inline` in the header so tests +and the registry can read it without linking the engine: + +```cpp +inline flm::backend::BackendTraits rai_traits() { + flm::backend::BackendTraits traits; + traits.needs_npu_xclbin = false; // no xclbin manager is built + traits.supports_preemption = false; // rejected before the factory runs + traits.max_context_length = 4096; // rejected before the factory runs + return traits; +} +``` + +### Register it + +One line in [`builtin_backends.cpp`](../AutoModel/builtin_backends.cpp), under the +guard: + +```cpp +#if defined(FLM_ENABLE_RAI) + registry.register_backend("", flm::backend::kRaiBackendId, + flm::::rai_factory(), + flm::::rai_traits()); +#endif +``` + +This is the only file that knows both family names and engine types. + +--- + +## 5. The frontend (new families only) + +If the family already exists, you are done with C++ — `AutoModel::_shared_load_backend` +resolves the id, checks the traits, builds your backend and points `lm_engine` at +its engine. No frontend change. + +For a new family, add `modeling_.{hpp,cpp}` under `AutoModel/` and wire it +into [`all_models.hpp`](../../include/AutoModel/all_models.hpp) — the enum, the +`modelFamilyMap` entry, and the `switch` case. Keep it backend-agnostic: +[`modeling_phi4.cpp`](../AutoModel/modeling_phi4.cpp) has no `#if +FLM_ENABLE_RAI` anywhere. It loads the backend, sets up the tokenizer +(honouring `backend_->forced_eos_ids()`), applies the sampler, and delegates the +decode loop to `_shared_generate`. + +Also register the FLM-side engine if the family has one: +`RegisterFlm<_npu>(registry, "")`. + +--- + +## 6. The catalog + +Two files, and both must agree. + +**[`model_list.json`](../../model_list.json)** — the entry. Phi-4 shares one tag +across both NPU generations, so the aie_next artifacts arrive as a +`platform_overrides.aie_next` patch: + +```jsonc +"": { + "": { + "supported_platforms": ["stx", "aie_next"], + "platform_overrides": { + "aie_next": { + "name": "", + "url": "...", "file_url": "...", "size": 4100140571, + "default_context_length": 4096, + "files": [".gguf", "tokenizer.json", "tokenizer_config.json", "config.json"], + "file_sources": { "tokenizer.json": { "url": "...", "revision": "..." } }, + "model_info_key": "-rai:", + "ms_url": null + } + } + } +} +``` + +Points that are easy to get wrong: + +- The patch is a **JSON merge-patch**: arrays replace wholesale, and `null` + *deletes* a key — that is what `"ms_url": null` is doing. +- `supported_platforms` is pruned at load, and **only aie_next support needs a + tag**: an entry that omits the key is stx-only, which is the overwhelming + majority. Do not write `["stx"]`; it restates the default. +- **The entry names no backend.** There is no `supported_backends` and no + `details.execution_backend` — both are retired. By the time an entry reaches + the loader it has already been filtered to the platform this build targets, + and the build links exactly one kernel flow, so there is nothing left for the + entry to decide. `--backend` and `FLM_BACKEND` override that default, and are + checked against the registry, not against the entry. +- `file_sources` pins a per-file origin + revision when the weights and the + tokenizer come from different repos (very common with GGUF mirrors). +- `model_info_key` redirects the downloader to a differently-named record set, + which is needed exactly because the tag is shared across platforms. + +**[`model_info.json`](../../model_info.json)** — one record per file, with `size` +and `sha256`. The downloader refuses anything it cannot match +([`model_downloader.cpp:46`](../../pull/model_downloader.cpp#L46)). + +--- + +## 7. Tests + +Three tiers, and the first two do not need an NPU: + +- **Pure logic** — the GGUF and host layers. Shape rejection, metadata parsing, + contract cross-validation, RMS norm and RoPE against reference values. +- **Frontend against a stub backend** — see + [`test/phi4_rai/test_phi4_frontend.cpp`](../../test/phi4_rai/test_phi4_frontend.cpp). + It defines its own empty `flm::backend::register_builtin_backends` (so it never + links the prebuilt engine libraries) and installs stubs with + `BackendRegistry::replace_backend`. The stub repeats the real backend's + pre-device work verbatim, which keeps the "nothing is opened before validation" + and "no device is created for a bad package" assertions meaningful. +- **Registry and selection** — [`test/model_backend/`](../../test/model_backend/), + Linux-buildable: registration, duplicate ids, resolution precedence, and the + error text for an id this build does not have. + +Add a `test/_rai/` directory following the Phi-4 one. Note the +trick it uses at configure time: it synthesizes a version-bumped copy of +`ryzenai/corelib.h` and asserts the adapter **fails** to compile against it, which +is how the version pin is actually enforced. + +--- + +## 8. Verify + +```powershell +flm pull : +flm run : --backend rai +flm run : --backend bogus # must exit 1, listing the real ids +``` + +Check, in order: + +1. `show_profile` prints `Backend: rai` and a `Backend detail:` line + with the loaded corelib path. +2. Generation is coherent. Garbage output on rai is almost always a shape-plan or + requantization-threading bug, not a tokenizer bug. +3. Decode stops at the window limit without a corelib error. +4. The prompt-length ceiling and the preemption rejection fire *before* any device + allocation — they are traits, checked by the frontend. +5. `git diff --stat` shows **zero** changes to `causal_lm.hpp`, `buffer.hpp`, + `tensor_2d.hpp`, `lm_config.hpp`, `q4_npu_eXpress.hpp` and `npu_utils/`. + A violation here fails at runtime, not at build time, which is why it is a + checklist item rather than a compiler's job. + +--- + +## Checklist + +- [ ] `src/common/models//rai/` created (no CMake edit) +- [ ] GGUF/host layers hold no corelib types +- [ ] `load_weights` is a documented throwing shim +- [ ] weight creates are serialized, with the thread hint passed +- [ ] runtime `shared_ptr` declared before the engine member +- [ ] all validation happens before `GetOrCreate` +- [ ] `BackendTraits` is `inline` in the header +- [ ] registered in `builtin_backends.cpp` under `#if defined(FLM_ENABLE_RAI)` +- [ ] no `#if FLM_ENABLE_RAI` anywhere in the frontend +- [ ] `model_list.json` + `model_info.json` agree, including `model_info_key` +- [ ] frozen headers untouched diff --git a/src/common/models/models_sources.cmake b/src/common/models/models_sources.cmake new file mode 100644 index 000000000..41370118d --- /dev/null +++ b/src/common/models/models_sources.cmake @@ -0,0 +1,7 @@ +# Per-model sources, split by where the kernels come from. A model that reaches +# its kernels through ryzenai-corelib puts them in: +# /rai/ - built into flm_rai when FLM_ENABLE_RAI is on +# FastFlowLM's own flow has no per-model sources here: those engines ship as +# prebuilt libraries under lib//. +# Adding a model means adding the folder, not editing this file. +file(GLOB FLM_MODELS_RAI_SOURCES "${CMAKE_CURRENT_LIST_DIR}/*/rai/*.cpp") diff --git a/src/common/models/phi4/rai/phi4_rai.cpp b/src/common/models/phi4/rai/phi4_rai.cpp new file mode 100644 index 000000000..18652dedf --- /dev/null +++ b/src/common/models/phi4/rai/phi4_rai.cpp @@ -0,0 +1,495 @@ +#include "models/phi4/rai/phi4_rai.hpp" +#include "rai/corelib_object.hpp" +#include "models/phi4/rai/phi4_rai_constants.hpp" +#include "models/phi4/rai/phi4_rai_host.hpp" +#include "models/phi4/rai/phi4_rai_shape_plan.hpp" +#include "models/phi4/rai/phi4_rai_weight_cache.hpp" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace flm::phi4 { +namespace { +using namespace flm::corelib; +std::string Name(std::size_t i, const char* suffix) { + return "blk." + std::to_string(i) + suffix; +} + +/// Load-time phase accounting. Model load on this backend is dominated by +/// requantizing every weight from Q8_0, and without a breakdown there is no way +/// to tell that from disk I/O or from shape planning. Set FLM_RAI_PROFILE_LOAD +/// to print it; the timer itself always runs, it costs five clock reads. +struct LoadPhases { + std::chrono::steady_clock::time_point mark{std::chrono::steady_clock::now()}; + double shape_plan{}, tensor_resolve{}, host_prep{}, weight_create{}, device_tensors{}; + double cache_write{}; + std::uint64_t cache_reclaimed{}; + bool from_cache{}; + + double Lap() { + const auto now = std::chrono::steady_clock::now(); + const double seconds = std::chrono::duration(now - mark).count(); + mark = now; + return seconds; + } + + void Report() const { + const char* enabled = std::getenv("FLM_RAI_PROFILE_LOAD"); + if (!enabled || !*enabled || *enabled == '0') return; + const double total = shape_plan + tensor_resolve + host_prep + + weight_create + device_tensors + cache_write; + std::ostringstream out; + out << std::fixed << std::setprecision(2) + << "[FLM] rai load: " << total << " s total" + << " (shape plan " << shape_plan + << ", GGUF resolve " << tensor_resolve + << ", host prep " << host_prep + << ", weight requantize " << weight_create + << ", device tensors " << device_tensors << ")" + << (from_cache ? " [weights from cache]" + : (cache_write > 0.0 ? " [cache written in " + + [&]{ std::ostringstream w; w << std::fixed + << std::setprecision(2) << cache_write; return w.str(); }() + " s]" + : std::string())) + << (cache_reclaimed > 0 + ? " [reclaimed " + std::to_string(cache_reclaimed / (1024 * 1024)) + + " MB of stale cache]" + : std::string()); + std::cout << out.str() << std::endl; + } +}; +} + +struct phi4_rai::Impl { + std::shared_ptr package; + std::shared_ptr runtime; + std::shared_ptr api; + // Declared before `plan` so it starts before the initializer list builds it. + LoadPhases phases; + Phi4ShapePlan plan; + std::uint32_t max_length; + int position{}; + std::optional saved; + bool poisoned{}; + UniqueStream stream; + std::array q_weights, k_weights, v_weights, o_weights; + std::array mlp_weights; + UniqueMatMulWeights lm_weights; + UniqueTensor hidden, residual, skip, q, k, attention, lm_input, logits, cosine, sine; + std::array k_cache, v_cache; + TensorView embedding; + FloatTensorView first_norm_scale; + + Impl(LM_Config, std::shared_ptr pkg, + std::shared_ptr rt, std::uint32_t maximum) + : package(std::move(pkg)), runtime(std::move(rt)), + api(runtime ? runtime->api() : nullptr), + max_length(maximum) { + if (!package) throw std::invalid_argument("Phi-4 GGUF package is null"); + if (!runtime || !api) throw std::invalid_argument("corelib runtime is null"); + if (!maximum || maximum > kMaxSequenceLength) + throw std::invalid_argument("Phi-4 maximum length must be in 1..4096"); + // The shape plan used to be built here, in the initializer list. Since + // 0.5.0 the padding helpers take the stream, so the plan cannot exist + // before one does and is built further down instead. + + // Validate and capture every mapped span before the first device create. + embedding = package->RequireQ8("token_embd.weight", std::array{kVocabularySize,kHiddenSize}); + auto final_norm = package->RequireF32("output_norm.weight", std::array{kHiddenSize}); + std::array an, fn; + std::array qkv, gu; + std::array ow, dw; + for (std::size_t i=0;iRequireF32(Name(i,".attn_norm.weight"),std::array{kHiddenSize}); + fn[i]=package->RequireF32(Name(i,".ffn_norm.weight"),std::array{kHiddenSize}); + qkv[i]=package->AttentionQkv(i); gu[i]=package->GateUp(i); + ow[i]=package->RequireQ8(Name(i,".attn_output.weight"),std::array{kHiddenSize,kHiddenSize}); + dw[i]=package->RequireQ8(Name(i,".ffn_down.weight"),std::array{kHiddenSize,kIntermediateSize}); + } + phases.tensor_resolve = phases.Lap(); + std::optional factors; + try { factors=package->RequireF32("rope_factors_short.weight",std::array{48}); } + catch (const std::runtime_error&) {} + auto rope=BuildShortRopeTables(package->Metadata(),factors); + auto final_bf=ConvertF32ToBf16(final_norm.values); + std::array,kLayerCount> an_bf,fn_bf; + for(std::size_t i=0;i epsf{kRmsEpsilon}; auto eps=ConvertF32ToBf16(epsf); + + first_norm_scale = an[0]; + phases.host_prep = phases.Lap(); + auto lease=runtime->AcquireExecution(); void* raw=nullptr; + api->Check(api->functions().create_stream(kPrefillPdi,kTokenPdi,&raw),"ryzenai_corelib_create_stream"); stream=UniqueStream(api,raw); + // Now that the stream exists the padding helpers can be asked. Every + // extent below is therefore an extent *for this stream's PDI pair*. + plan=Phi4ShapePlan::Build(api,stream.get()); + phases.shape_plan = phases.Lap(); + auto mm=[&](const TensorView& tv,std::int64_t kk,std::int64_t nn,const std::string& label){ + ryzenai_corelib_matmul_bf16_weights_desc d{kk,nn,kRequantizedGroupSize,false}; + ryzenai_corelib_matmul_bf16_gguf_components c{tv.bytes.data(),ryzenai_corelib_gguf_quant_type_q8_0}; void* p=nullptr; + api->Check(api->functions().matmul_weights_create_gguf_requantized(&d,&c,kRequantizeThreads,&p),"ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized "+label); + return UniqueMatMulWeights(api,p); + }; + // Each create reads its own mapped range and writes its own array slot, + // so they are independent. Collect them first, then run the list across + // a small pool -- packing is the whole of model load and it is the one + // phase with nothing to serialize. + std::vector> creates; + creates.reserve(kLayerCount * 5 + 1); + for(std::size_t i=0;iCheck(api->functions().ssmlp_weights_create_gguf_requantized(&d,&c,kRequantizeThreads,&p),"ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized layer "+std::to_string(i)); + mlp_weights[i]=UniqueSsMlpWeights(api,p); + }); + } + creates.push_back([&]{ lm_weights=mm(embedding,kHiddenSize,kVocabularySize,"token_embd.weight"); }); + + std::atomic next_create{0}; + std::mutex failure_mutex; + std::exception_ptr first_failure; + const auto run_creates=[&]{ + for(std::size_t k=next_create++;k lock(failure_mutex); + if(!first_failure) first_failure=std::current_exception(); + // Stop the others: the first diagnostic is the useful one, + // and a package that fails one create fails the load. + next_create=creates.size(); + return; + } + } + }; + // A cache hit replaces every create above with a mapped slice, which is + // the whole point: the refit is paid once per GGUF rather than once per + // launch. A miss, a stale key or any failure just packs as normal. + const auto cache_directory = WeightCacheDirectory(package->Path().parent_path()); + const auto cache_key = cache_directory + ? std::optional(MakeWeightCacheKey( + package->Path(), api->runtime_version().major, + api->runtime_version().minor, api->runtime_version().patch, + kRequantizedGroupSize, creates.size())) + : std::nullopt; + bool loaded_from_cache = false; + if (cache_directory && cache_key) { + if (const auto index = ReadWeightCacheIndex(*cache_directory, *cache_key)) { + loaded_from_cache = LoadWeightsFromCache( + WeightCacheDataPath(*cache_directory), index->spans); + } + } + + if (!loaded_from_cache) { + const std::size_t workers=std::min(kWeightCreateConcurrency,creates.size()); + std::vector pool; + pool.reserve(workers>0?workers-1:0); + for(std::size_t t=1;t dims,const char* label){ + std::vector shape(dims);void* p=nullptr; + api->Check(api->functions().create_device_tensor(type,shape.data(),shape.size(),&p),std::string("ryzenai_corelib_create_device_tensor ")+label); + return UniqueTensor(api,p); + }; + hidden=tensor(ryzenai_corelib_data_type_bf16,{rows,kHiddenSize},"hidden"); + residual=tensor(ryzenai_corelib_data_type_bf16,{rows,kHiddenSize},"residual"); + skip=tensor(ryzenai_corelib_data_type_bf16,{rows,kHiddenSize},"skip"); + q=tensor(ryzenai_corelib_data_type_bf16,{query_rows,kQueryDimension},"query"); + k=tensor(ryzenai_corelib_data_type_bf16,{key_rows,kKvDimension},"key"); + attention=tensor(ryzenai_corelib_data_type_bf16,{attention_rows,kQueryDimension},"attention"); + lm_input=tensor(ryzenai_corelib_data_type_bf16,{1,kHiddenSize},"lm input"); + logits=tensor(ryzenai_corelib_data_type_bf16,{1,kVocabularySize},"logits"); + cosine=tensor(ryzenai_corelib_data_type_fp32,{kMaxSequenceLength,48},"cosine"); + sine=tensor(ryzenai_corelib_data_type_fp32,{kMaxSequenceLength,48},"sine"); + for(std::size_t i=0;iCheck(api->functions().tensor_write(cosine.get(),ryzenai_corelib_data_type_fp32,rope.cosine.data(),rope.cosine.size(),0),"ryzenai_corelib_tensor_write cosine"); + api->Check(api->functions().tensor_write(sine.get(),ryzenai_corelib_data_type_fp32,rope.sine.data(),rope.sine.size(),0),"ryzenai_corelib_tensor_write sine"); + phases.device_tensors = phases.Lap(); + phases.Report(); + } + + /// \brief the weight slot order the cache is written and read in + /// \param slot 0..160: five per layer, then the LM head + /// \note Both directions walk this same order, so an index entry always + /// refers to the weight it was written from. The creates complete out + /// of order, but they are *assigned* by this index, not by completion. + static bool SlotIsMatmul(std::size_t slot) { return slot % 5 != 4 || slot == kLayerCount * 5; } + + ryzenai_corelib_matmul_bf16_weights_desc MatmulDescAt(std::size_t slot) const { + if (slot == kLayerCount * 5) + return {kHiddenSize, kVocabularySize, kRequantizedGroupSize, false}; + switch (slot % 5) { + case 0: return {kHiddenSize, kQueryDimension, kRequantizedGroupSize, false}; + case 1: + case 2: return {kHiddenSize, kKvDimension, kRequantizedGroupSize, false}; + default: return {kHiddenSize, kHiddenSize, kRequantizedGroupSize, false}; + } + } + + void* WeightHandleAt(std::size_t slot) const { + if (slot == kLayerCount * 5) return lm_weights.get(); + const auto layer = slot / 5; + switch (slot % 5) { + case 0: return q_weights[layer].get(); + case 1: return k_weights[layer].get(); + case 2: return v_weights[layer].get(); + case 3: return o_weights[layer].get(); + default: return mlp_weights[layer].get(); + } + } + + void AssignWeightAt(std::size_t slot, void* handle) { + if (slot == kLayerCount * 5) { lm_weights = UniqueMatMulWeights(api, handle); return; } + const auto layer = slot / 5; + switch (slot % 5) { + case 0: q_weights[layer] = UniqueMatMulWeights(api, handle); break; + case 1: k_weights[layer] = UniqueMatMulWeights(api, handle); break; + case 2: v_weights[layer] = UniqueMatMulWeights(api, handle); break; + case 3: o_weights[layer] = UniqueMatMulWeights(api, handle); break; + default: mlp_weights[layer] = UniqueSsMlpWeights(api, handle); break; + } + } + + /// \brief bind every weight from the cache file instead of packing + /// \return true when all of them loaded; false leaves nothing bound + /// \note Any failure abandons the whole attempt rather than packing the + /// remainder: a half-cached model is not a state worth having, and + /// the caller simply packs. corelib rejects a slice that is not + /// exactly what the descriptor packs to, so a stale file is caught + /// here rather than becoming confident nonsense. + bool LoadWeightsFromCache(const std::filesystem::path& data_path, + const std::vector& spans) { + const auto path = data_path.string(); + const std::size_t total = kLayerCount * 5 + 1; + if (spans.size() != total) return false; + for (std::size_t slot = 0; slot < total; ++slot) { + void* handle = nullptr; + ryzenai_corelib_status status; + if (SlotIsMatmul(slot)) { + const auto desc = MatmulDescAt(slot); + status = api->functions().matmul_weights_create_from_file( + &desc, path.c_str(), spans[slot].offset, spans[slot].size, &handle); + } else { + const ryzenai_corelib_ssmlp_bf16_weights_desc desc{ + kHiddenSize, kIntermediateSize, kRequantizedGroupSize}; + status = api->functions().ssmlp_weights_create_from_file( + &desc, path.c_str(), spans[slot].offset, spans[slot].size, &handle); + } + if (status != ryzenai_corelib_status_success || handle == nullptr) { + ReleaseAllWeights(); + return false; + } + AssignWeightAt(slot, handle); + } + return true; + } + + void ReleaseAllWeights() { + for (std::size_t layer = 0; layer < kLayerCount; ++layer) { + q_weights[layer] = {}; k_weights[layer] = {}; + v_weights[layer] = {}; o_weights[layer] = {}; + mlp_weights[layer] = {}; + } + lm_weights = {}; + } + + /// \brief copy the packed bytes out and write them beside the model + /// \note Best effort: a cache that cannot be written is not a load failure, + /// it only means the next launch packs again. The index is written + /// last, so a data file without a matching index is never used. + void WriteWeightCache(const std::filesystem::path& directory, + const WeightCacheKey& key) { + try { + std::error_code error; + std::filesystem::create_directories(directory, error); + const auto data_path = WeightCacheDataPath(directory); + const auto temporary = data_path.string() + ".tmp"; + std::vector spans; + spans.reserve(kLayerCount * 5 + 1); + { + std::ofstream output(temporary, std::ios::binary | std::ios::trunc); + if (!output) return; + std::vector buffer; + std::uint64_t offset = 0; + for (std::size_t slot = 0; slot < kLayerCount * 5 + 1; ++slot) { + std::size_t size = 0; + if (api->functions().weights_copy_data( + WeightHandleAt(slot), nullptr, 0, &size) != + ryzenai_corelib_status_success || size == 0) return; + buffer.resize(size); + if (api->functions().weights_copy_data( + WeightHandleAt(slot), buffer.data(), buffer.size(), &size) != + ryzenai_corelib_status_success) return; + output.write(buffer.data(), static_cast(size)); + if (!output) return; + spans.push_back({offset, static_cast(size)}); + offset += size; + } + } + std::filesystem::rename(temporary, data_path, error); + if (error) { std::filesystem::remove(temporary, error); return; } + if (!WriteWeightCacheIndex(directory, key, spans)) { + // No index means the data file would never be used; drop it + // rather than leave the space occupied. + std::filesystem::remove(data_path, error); + } + } catch (...) { + // Caching is an optimisation; never let it fail a load. + } + } + + void usable() const {if(poisoned)throw std::runtime_error("Phi-4 corelib engine is poisoned");} + buffer run(std::span ids,bool prefill){ + usable(); if(ids.empty())throw std::invalid_argument("Phi-4 request contains no token IDs"); + if(prefill&&position)throw std::runtime_error("Phi-4 prefill must start at logical position zero"); + if(ids.size()>max_length||position+ids.size()>max_length||position+ids.size()>kMaxSequenceLength)throw std::out_of_range("Phi-4 request exceeds configured context capacity"); + if(!prefill&&position+ids.size()>kMaxDecodeWindow)throw std::out_of_range("Phi-4 decode window stops at position 4095"); + auto decoded=DecodeEmbeddingRowsQ8(embedding,ids);const auto&e=plan.ForRows(ids.size()); + auto rows=std::max({e.query_rows,e.kv_rows,e.output_rows,e.ssmlp_rows}); + std::vector normalized(decoded.size()); + HostRmsNorm(decoded,first_norm_scale.values,ids.size(),kHiddenSize, + kRmsEpsilon,normalized); + std::vector input(static_cast(rows*kHiddenSize),0); + std::vector residual_input(static_cast(rows*kHiddenSize),0); + std::copy(normalized.begin(),normalized.end(),input.begin()); + std::copy(decoded.begin(),decoded.end(),residual_input.begin()); + + auto lease=runtime->AcquireExecution();bool submitted=false; + try{ + api->Check(api->functions().tensor_write(hidden.get(),ryzenai_corelib_data_type_fp32,input.data(),input.size(),0),"ryzenai_corelib_tensor_write hidden"); + api->Check(api->functions().tensor_write(residual.get(),ryzenai_corelib_data_type_fp32,residual_input.data(),residual_input.size(),0),"ryzenai_corelib_tensor_write residual embedding"); + // 0.5.0 took the row count out of every dispatch: M now comes from + // the operand's own shape, and each operand is checked against its + // OWN padded extent rather than resolved through the allocation it + // sits in. So each op sees the tensors through a window at the + // extent the plan interrogated for it. Built once here, not per + // layer: ids.size() does not change inside the loop, and the + // allocations themselves still happen only at load. + const auto& extents=plan.ForRows(ids.size()); + auto view=[&](void* parent,std::int64_t rows,std::int64_t cols,const char* label){ + std::array shape{rows,cols};void* p=nullptr; + api->Check(api->functions().create_tensor_window(parent,shape.data(),shape.size(),0,&p),std::string("ryzenai_corelib_create_tensor_window ")+label); + return UniqueTensorWindow(api,p); + }; + const auto hidden_q=view(hidden.get(),extents.query_rows,kHiddenSize,"hidden@query"); + const auto hidden_kv=view(hidden.get(),extents.kv_rows,kHiddenSize,"hidden@kv"); + const auto hidden_out=view(hidden.get(),extents.output_rows,kHiddenSize,"hidden@output"); + const auto hidden_mlp=view(hidden.get(),extents.ssmlp_rows,kHiddenSize,"hidden@ssmlp"); + const auto q_mm=view(q.get(),extents.query_rows,kQueryDimension,"query@matmul"); + const auto q_mha=view(q.get(),extents.flat_mha_rows,kQueryDimension,"query@mha"); + const auto k_mm=view(k.get(),extents.kv_rows,kKvDimension,"key@matmul"); + const auto k_mha=view(k.get(),extents.flat_mha_rows,kKvDimension,"key@mha"); + const auto attention_mha=view(attention.get(),extents.flat_mha_rows,kQueryDimension,"attention@mha"); + const auto attention_mm=view(attention.get(),extents.output_rows,kQueryDimension,"attention@matmul"); + const auto residual_mlp=view(residual.get(),extents.ssmlp_rows,kHiddenSize,"residual@ssmlp"); + const auto skip_mlp=view(skip.get(),extents.ssmlp_rows,kHiddenSize,"skip@ssmlp"); + void* res=residual_mlp.get();void* sk=skip_mlp.get(); + for(std::size_t i=0;ifunctions().matmul( + stream.get(),hidden_q.get(),q_weights[i].get(),q_mm.get()); + submitted=submitted || query_status==ryzenai_corelib_status_success || + query_status==ryzenai_corelib_status_failure; + api->Check(query_status,"ryzenai_corelib_matmul_bf16 query layer "+std::to_string(i)); + api->Check(api->functions().matmul(stream.get(),hidden_kv.get(),k_weights[i].get(),k_mm.get()),"ryzenai_corelib_matmul_bf16 key layer "+std::to_string(i)); + // The gm exception: a cache write takes its position from the + // bound buffer's base, so this window stays a position, not an + // extent -- its height is what bounds the rows that land there. + std::array shape{8,kMaxSequenceLength-position,128};void* p=nullptr; + api->Check(api->functions().create_tensor_window(v_cache[i].get(),shape.data(),shape.size(),static_cast(position)*128,&p),"ryzenai_corelib_create_tensor_window V");UniqueTensorWindow win(api,p); + api->Check(api->functions().matmul(stream.get(),hidden_kv.get(),v_weights[i].get(),win.get()),"ryzenai_corelib_matmul_bf16 value layer "+std::to_string(i)); + api->Check(api->functions().flat_mha(stream.get(),&plan.attention_desc(),q_mha.get(),k_mha.get(),position,cosine.get(),sine.get(),k_cache[i].get(),v_cache[i].get(),attention_mha.get()),"ryzenai_corelib_flat_mha_bf16 layer "+std::to_string(i)); + api->Check(api->functions().matmul(stream.get(),attention_mm.get(),o_weights[i].get(),hidden_out.get()),"ryzenai_corelib_matmul_bf16 output layer "+std::to_string(i)); + api->Check(api->functions().ssmlp(stream.get(),hidden_mlp.get(),res,mlp_weights[i].get(),sk,hidden_mlp.get()),"ryzenai_corelib_ssmlp_bf16 layer "+std::to_string(i));std::swap(res,sk); + } + api->Check(api->functions().stream_synchronize(stream.get()),"ryzenai_corelib_stream_synchronize hidden"); + std::vector row(kHiddenSize);api->Check(api->functions().tensor_read(hidden.get(),ryzenai_corelib_data_type_bf16,row.data(),row.size(),(ids.size()-1)*kHiddenSize),"ryzenai_corelib_tensor_read final hidden row"); + api->Check(api->functions().tensor_write(lm_input.get(),ryzenai_corelib_data_type_bf16,row.data(),row.size(),0),"ryzenai_corelib_tensor_write LM head input"); + // lm_input and logits are allocated one row tall, which is exactly + // how 0.5.0 is told to select the token PDI for the head. + api->Check(api->functions().matmul(stream.get(),lm_input.get(),lm_weights.get(),logits.get()),"ryzenai_corelib_matmul_bf16 LM head"); + api->Check(api->functions().stream_synchronize(stream.get()),"ryzenai_corelib_stream_synchronize logits"); + buffer out(kVocabularySize);api->Check(api->functions().tensor_read(logits.get(),ryzenai_corelib_data_type_bf16,out.data(),out.size(),0),"ryzenai_corelib_tensor_read logits");position+=static_cast(ids.size());return out; + }catch(...){if(submitted){(void)api->functions().stream_synchronize(stream.get());poisoned=true;position=0;saved.reset();}throw;} + } + buffer read_cache(bool is_k,int layer,int index){ + usable(); + if(layer<0||layer>=kLayerCount||index<0||index>=kMaxSequenceLength) + throw std::out_of_range("Phi-4 cache index is out of range"); + auto lease=runtime->AcquireExecution(); + api->Check(api->functions().stream_synchronize(stream.get()), + "ryzenai_corelib_stream_synchronize cache read"); + buffer out(kKvHeadCount*kHeadSize); + void* cache=is_k?k_cache[layer].get():v_cache[layer].get(); + for(std::size_t head=0;head(index))*kHeadSize; + api->Check(api->functions().tensor_read( + cache,ryzenai_corelib_data_type_bf16, + out.data()+head*kHeadSize,kHeadSize,offset), + "ryzenai_corelib_tensor_read cache head "+ + std::to_string(head)); + } + return out; + } +}; + +phi4_rai::phi4_rai(LM_Config c,std::shared_ptr p,std::shared_ptr r,std::uint32_t m):impl_(std::make_unique(std::move(c),std::move(p),std::move(r),m)){} +phi4_rai::~phi4_rai()=default; +buffer phi4_rai::forward(int id){return impl_->run(std::span(&id,1),false);} +buffer phi4_rai::prefill(std::vector&ids,void*){return impl_->run(ids,true);} +void phi4_rai::set_context_length(int n){impl_->usable();if(n<0||static_cast(n)>impl_->max_length)throw std::out_of_range("Phi-4 context length is out of range");impl_->position=n;} +// An ABI shim, not a capability. load_weights is pure virtual in causal_lm.hpp, +// which is frozen because the engine libraries in src/lib/ are prebuilt +// against it. Nothing calls this: FlmBackend loads weights through the +// concrete engine type, and this engine's weights come from the GGUF package it +// was constructed with. See AutoModel/model_backend.hpp. +void phi4_rai::load_weights(Q4NX&){impl_->usable();throw std::runtime_error("Phi-4 rai weights are loaded only from GGUF");} +void phi4_rai::update_max_length(std::uint32_t n){impl_->usable();if(!n||n>kMaxSequenceLength||n(impl_->position))throw std::out_of_range("Phi-4 maximum length is invalid");impl_->max_length=n;} +void phi4_rai::clear_context(){impl_->usable();impl_->position=0;impl_->saved.reset();} +buffer phi4_rai::get_k_cache(int l,int i){return impl_->read_cache(true,l,i);} +buffer phi4_rai::get_v_cache(int l,int i){return impl_->read_cache(false,l,i);} +int phi4_rai::get_current_context_length(){impl_->usable();return impl_->position;} +int phi4_rai::checkpoint(){impl_->usable();impl_->saved=impl_->position;return impl_->position;} +int phi4_rai::restore(){impl_->usable();if(!impl_->saved)return -1;return impl_->position=*impl_->saved;} +bool phi4_rai::poisoned()const noexcept{return impl_&&impl_->poisoned;} +} // namespace flm::phi4 diff --git a/src/common/models/phi4/rai/phi4_rai_backend.cpp b/src/common/models/phi4/rai/phi4_rai_backend.cpp new file mode 100644 index 000000000..3cd58f7ac --- /dev/null +++ b/src/common/models/phi4/rai/phi4_rai_backend.cpp @@ -0,0 +1,131 @@ +/// \file phi4_rai_backend.cpp +/// \brief The ryzenai-corelib backend for Phi-4 +#include "models/phi4/rai/phi4_rai_backend.hpp" + +#include "rai/corelib_runtime.hpp" +#include "models/phi4/rai/phi4_rai.hpp" +#include "models/phi4/rai/phi4_rai_gguf.hpp" +#include "utils/file_access.hpp" +#include "utils/utils.hpp" + +#include +#include +#include +#include +#include +#include +#include + +namespace flm::phi4 { +namespace { + +/// \brief the only GGUF this backend accepts; there is deliberately no alias +constexpr const char* kRaiGguf = "Phi-4-mini-instruct.Q8_0.gguf"; + +/// \brief the EOS ids ValidatePhi4Contract proves against the GGUF +/// \note These come from three independent sources agreeing, which is a +/// stronger guarantee than tokenizer_config.json alone. +const std::vector kRaiEosIds = {200020, 199999}; + +/// \brief read a JSON file, recording the open for the file-access audit +/// \param path the file +/// \return the parsed document +nlohmann::json ReadJson(const std::filesystem::path& path) { + flm::file_access::ObserveOpen(path); + std::ifstream input(path, std::ios::binary); + if (!input) throw std::runtime_error("Cannot open " + path.string()); + try { + return nlohmann::json::parse(input); + } catch (const std::exception& error) { + throw std::runtime_error("Cannot parse " + path.string() + ": " + + error.what()); + } +} + +/// \brief Phi-4 on ryzenai-corelib, GGUF weights +class RaiBackend final : public flm::backend::ModelBackend { +public: + explicit RaiBackend(const flm::backend::BackendContext& context) { + if (!context.config) { + throw std::runtime_error("Phi-4 rai backend needs an LM_Config"); + } + if (context.enable_preemption) { + throw std::invalid_argument("Phi-4 rai does not support preemption"); + } + if (context.context_length < 1 || + context.context_length > kRaiContextLimit) { + throw std::out_of_range("Phi-4 rai context length must be in 1..4096"); + } + + // Read and validate every source of truth before acquiring the runtime + // or creating the engine, so a mismatched package fails while nothing + // has been allocated on the device. + const std::filesystem::path root(context.model_path); + const auto config = ReadJson(root / "config.json"); + const auto tokenizer_json = ReadJson(root / "tokenizer.json"); + // The frontend already parsed this one and passes it down, so the file + // is opened once per load and the directory layout stays its knowledge. + if (context.tokenizer_config == nullptr) { + throw std::runtime_error( + "Phi-4 rai backend needs the frontend's tokenizer_config.json"); + } + auto package = Phi4GgufPackage::Open(root / kRaiGguf); + package->ValidatePhi4Contract(config, tokenizer_json, + *context.tokenizer_config); + + runtime_ = corelib::CorelibRuntime::GetOrCreate( + utils::get_executable_directory()); + auto engine = std::make_unique( + *context.config, std::move(package), runtime_, + context.context_length); + engine->clear_context(); + engine_ = std::move(engine); + } + + ~RaiBackend() override { + // The engine holds its own reference to the runtime, but destroy it + // first anyway: corelib objects must not outlive the API they came from. + engine_.reset(); + } + + causal_lm& engine() override { return *engine_; } + + std::string id() const override { return flm::backend::kRaiBackendId; } + + std::string detail() const override { + return runtime_ ? runtime_->loaded_library_path().string() + : std::string(); + } + + std::uint32_t max_decode_length() const override { return kRaiDecodeLimit; } + + bool supports_preemption() const override { return false; } + + /// \note corelib rejects a decode past its own limit, so the extra forward() + /// the FastFlowLM engines want after an EOS token would fail here. + bool forwards_past_eos() const override { return false; } + + bool poisoned() const noexcept override { + return engine_ && engine_->poisoned(); + } + + std::optional> forced_eos_ids() const override { + return kRaiEosIds; + } + +private: + // Declared first so it outlives the engine. + std::shared_ptr runtime_; + std::unique_ptr engine_; +}; + +} // namespace + +flm::backend::BackendFactory rai_factory() { + return [](const flm::backend::BackendContext& context) + -> std::unique_ptr { + return std::make_unique(context); + }; +} + +} // namespace flm::phi4 diff --git a/src/common/models/phi4/rai/phi4_rai_gguf.cpp b/src/common/models/phi4/rai/phi4_rai_gguf.cpp new file mode 100644 index 000000000..278a52e0a --- /dev/null +++ b/src/common/models/phi4/rai/phi4_rai_gguf.cpp @@ -0,0 +1,658 @@ +#include "models/phi4/rai/phi4_rai_gguf.hpp" + +#include "models/phi4/rai/phi4_rai_constants.hpp" +#include "utils/file_access.hpp" + +#define NOMINMAX +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace flm::phi4 { +namespace { +constexpr std::uint32_t kMagic = 0x46554747; +constexpr std::uint32_t kVersion = 3; +constexpr std::uint32_t kTypeF32 = 0; +constexpr std::uint32_t kTypeQ8_0 = 8; + +[[noreturn]] void Fail(std::string_view field, std::string actual, + std::string expected) { + throw std::runtime_error(std::string(field) + ": actual " + actual + + ", expected " + expected); +} + +std::uint64_t CheckedAdd(std::uint64_t a, std::uint64_t b, + std::string_view field) { + if (a > std::numeric_limits::max() - b) + throw std::runtime_error(std::string(field) + ": overflow in addition"); + return a + b; +} + +std::uint64_t CheckedMultiply(std::uint64_t a, std::uint64_t b, + std::string_view field) { + if (a != 0 && b > std::numeric_limits::max() / a) + throw std::runtime_error(std::string(field) + ": overflow in product"); + return a * b; +} + +std::uint64_t AlignUp(std::uint64_t value, std::uint64_t alignment) { + if (alignment == 0 || (alignment & (alignment - 1)) != 0) + Fail("general.alignment", std::to_string(alignment), "a non-zero power of two"); + return CheckedAdd(value, alignment - 1, "alignment") & ~(alignment - 1); +} + +std::span RequireRange(std::span file, + std::uint64_t offset, + std::uint64_t length, + std::string_view field) { + const auto end = CheckedAdd(offset, length, field); + if (end > file.size() || offset > std::numeric_limits::max() || + length > std::numeric_limits::max()) + Fail(field, "out-of-file range", "range within mapped file"); + return file.subspan(static_cast(offset), + static_cast(length)); +} + +class Cursor { +public: + Cursor(std::span file, std::uint64_t offset = 0) + : file_(file), offset_(offset) {} + + template + T Read(std::string_view field) { + const auto bytes = RequireRange(file_, offset_, sizeof(T), field); + T value; + std::memcpy(&value, bytes.data(), sizeof(T)); + offset_ = CheckedAdd(offset_, sizeof(T), field); + return value; + } + + std::string ReadString(std::string_view field) { + const auto length = Read(field); + const auto bytes = RequireRange(file_, offset_, length, field); + std::string value(reinterpret_cast(bytes.data()), bytes.size()); + offset_ = CheckedAdd(offset_, length, field); + return value; + } + + void Skip(std::uint64_t length, std::string_view field) { + RequireRange(file_, offset_, length, field); + offset_ = CheckedAdd(offset_, length, field); + } + + std::uint64_t offset() const noexcept { return offset_; } + +private: + std::span file_; + std::uint64_t offset_; +}; + +std::string MetadataTypeName(std::uint32_t type) { + static constexpr const char* names[] = {"UINT8", "INT8", "UINT16", "INT16", + "UINT32", "INT32", "FLOAT32", "BOOL", "STRING", "ARRAY", "UINT64", + "INT64", "FLOAT64"}; + return type < std::size(names) ? names[type] : "unknown(" + std::to_string(type) + ")"; +} + +std::uint64_t FixedMetadataSize(std::uint32_t type) { + switch (type) { + case 0: case 1: case 7: return 1; + case 2: case 3: return 2; + case 4: case 5: case 6: return 4; + case 10: case 11: case 12: return 8; + default: return 0; + } +} + +struct ArrayInfo { std::uint32_t type; std::uint64_t count; }; +using MetadataValue = std::variant; + +MetadataValue ReadMetadataValue(Cursor& cursor, std::uint32_t type, + std::string_view field, bool retain) { + switch (type) { + case 0: { auto v = cursor.Read(field); return retain ? MetadataValue(std::uint64_t(v)) : MetadataValue{}; } + case 1: { auto v = cursor.Read(field); return retain ? MetadataValue(std::int64_t(v)) : MetadataValue{}; } + case 2: { auto v = cursor.Read(field); return retain ? MetadataValue(std::uint64_t(v)) : MetadataValue{}; } + case 3: { auto v = cursor.Read(field); return retain ? MetadataValue(std::int64_t(v)) : MetadataValue{}; } + case 4: { auto v = cursor.Read(field); return retain ? MetadataValue(std::uint64_t(v)) : MetadataValue{}; } + case 5: { auto v = cursor.Read(field); return retain ? MetadataValue(std::int64_t(v)) : MetadataValue{}; } + case 6: { auto v = cursor.Read(field); return retain ? MetadataValue(double(v)) : MetadataValue{}; } + case 7: { auto v = cursor.Read(field); if (v > 1) Fail(field, std::to_string(v), "GGUF boolean 0 or 1"); return retain ? MetadataValue(bool(v)) : MetadataValue{}; } + case 8: { auto v = cursor.ReadString(field); return retain ? MetadataValue(std::move(v)) : MetadataValue{}; } + case 9: { + const std::string array_field = std::string(field) + " array"; + const auto element_type = cursor.Read(array_field); + const auto count = cursor.Read(array_field); + if (element_type == 9 || element_type > 12) + Fail(array_field, MetadataTypeName(element_type), "a skippable GGUF array element type"); + const auto fixed = FixedMetadataSize(element_type); + if (fixed != 0) { + cursor.Skip(CheckedMultiply(count, fixed, array_field), array_field); + } else { + const auto minimum = CheckedMultiply(count, std::uint64_t{8}, array_field); + (void)minimum; + for (std::uint64_t i = 0; i < count; ++i) + (void)ReadMetadataValue(cursor, element_type, array_field, false); + } + return retain ? MetadataValue(ArrayInfo{element_type, count}) : MetadataValue{}; + } + case 10: { auto v = cursor.Read(field); return retain ? MetadataValue(v) : MetadataValue{}; } + case 11: { auto v = cursor.Read(field); return retain ? MetadataValue(v) : MetadataValue{}; } + case 12: { auto v = cursor.Read(field); return retain ? MetadataValue(v) : MetadataValue{}; } + default: + Fail(field, MetadataTypeName(type), "a supported metadata type"); + } +} + +bool IsRetainedKey(std::string_view key) { + static constexpr std::string_view keys[] = { + "general.architecture", "general.alignment", "phi3.block_count", + "phi3.context_length", "phi3.embedding_length", "phi3.feed_forward_length", + "phi3.attention.head_count", "phi3.attention.head_count_kv", + "phi3.attention.layer_norm_rms_epsilon", "phi3.rope.dimension_count", + "phi3.rope.freq_base", "phi3.rope.scaling.attn_factor", + "phi3.rope.scaling.original_context_length", "tokenizer.ggml.tokens", + "tokenizer.ggml.add_bos_token", "tokenizer.ggml.eos_token_id"}; + return std::find(std::begin(keys), std::end(keys), key) != std::end(keys); +} + +std::string ShapeText(std::span shape) { + std::ostringstream out; + out << '['; + for (std::size_t i = 0; i < shape.size(); ++i) { + if (i) out << ','; + out << shape[i]; + } + return out.str() + ']'; +} + +std::string GgmlTypeName(std::uint32_t type) { + if (type == kTypeF32) return "F32"; + if (type == kTypeQ8_0) return "Q8_0"; + return "GGML type " + std::to_string(type); +} + +std::uint64_t ElementCount(std::span shape, + std::string_view field) { + std::uint64_t result = 1; + for (const auto dimension : shape) { + if (dimension <= 0) Fail(field, std::to_string(dimension), "positive dimensions"); + result = CheckedMultiply(result, static_cast(dimension), field); + } + return result; +} + +std::uint64_t TensorByteLength(std::uint32_t type, + std::span shape, + std::string_view field) { + const auto elements = ElementCount(shape, field); + if (type == kTypeF32) return CheckedMultiply(elements, 4, field); + if (type == kTypeQ8_0) { + if (elements % 32 != 0) + Fail(field, std::to_string(elements) + " elements", "Q8_0 element count divisible by 32"); + return CheckedMultiply(elements / 32, 34, field); + } + Fail(field, GgmlTypeName(type), "F32 or Q8_0"); +} + +std::string JsonText(const nlohmann::json& value) { + return value.dump(); +} + +void RequireJsonString(const nlohmann::json& object, std::string_view key, + std::string_view expected) { + const auto it = object.find(std::string(key)); + if (it == object.end() || !it->is_string()) + Fail(key, it == object.end() ? "missing" : JsonText(*it), std::string(expected)); + const auto actual = it->get_ref(); + if (actual != expected) Fail(key, actual, std::string(expected)); +} + +void RequireJsonBoolean(const nlohmann::json& object, std::string_view key, + bool expected) { + const auto it = object.find(std::string(key)); + if (it == object.end() || !it->is_boolean()) + Fail(key, it == object.end() ? "missing" : JsonText(*it), expected ? "true" : "false"); + const auto actual = it->get(); + if (actual != expected) Fail(key, actual ? "true" : "false", expected ? "true" : "false"); +} + +void RequireJsonUnsigned(const nlohmann::json& object, std::string_view key, + std::uint64_t expected) { + const auto it = object.find(std::string(key)); + if (it == object.end()) Fail(key, "missing", std::to_string(expected)); + std::uint64_t actual; + if (it->is_number_unsigned()) { + actual = it->get(); + } else if (it->is_number_integer()) { + const auto signed_value = it->get(); + if (signed_value < 0) + Fail(key, JsonText(*it), "non-negative integer " + std::to_string(expected)); + actual = static_cast(signed_value); + } else { + Fail(key, JsonText(*it), "integer " + std::to_string(expected)); + } + if (actual != expected) Fail(key, std::to_string(actual), std::to_string(expected)); +} + +void RequireJsonDouble(const nlohmann::json& object, std::string_view key, + double expected) { + const auto it = object.find(std::string(key)); + if (it == object.end() || !it->is_number()) + Fail(key, it == object.end() ? "missing" : JsonText(*it), std::to_string(expected)); + const auto actual = it->get(); + if (!std::isfinite(actual) || actual != expected) + Fail(key, JsonText(*it), std::to_string(expected)); +} +} // namespace + +struct Phi4GgufPackage::Impl { + struct TensorRecord { + std::string name; + std::span bytes; + std::vector shape; + std::uint32_t type; + std::uint64_t absolute_offset; + }; + + std::filesystem::path path; + HANDLE file = INVALID_HANDLE_VALUE; + HANDLE mapping = nullptr; + const std::byte* data = nullptr; + std::uint64_t size = 0; + std::map> tensors; + std::map> metadata; + std::map> metadata_types; + + ~Impl() { + if (data) UnmapViewOfFile(data); + if (mapping) CloseHandle(mapping); + if (file != INVALID_HANDLE_VALUE) CloseHandle(file); + } + + std::span bytes() const { + return {data, static_cast(size)}; + } + + const TensorRecord& Tensor(std::string_view name) const { + const auto it = tensors.find(name); + if (it == tensors.end()) Fail(name, "missing", "present tensor"); + return it->second; + } + + std::uint64_t Unsigned(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "unsigned integer metadata"); + if (const auto* value = std::get_if(&it->second)) return *value; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "unsigned integer metadata"); + } + + double Number(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "floating-point metadata"); + if (const auto* value = std::get_if(&it->second)) return *value; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "floating-point metadata"); + } + + bool Boolean(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "boolean metadata"); + if (const auto* value = std::get_if(&it->second)) return *value; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "boolean metadata"); + } + + std::string String(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "string metadata"); + if (const auto* value = std::get_if(&it->second)) return *value; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "string metadata"); + } + + std::uint64_t ArrayCount(std::string_view key) const { + const auto it = metadata.find(key); + if (it == metadata.end()) Fail(key, "missing", "array metadata"); + if (const auto* value = std::get_if(&it->second)) return value->count; + Fail(key, MetadataTypeName(metadata_types.at(std::string(key))), "array metadata"); + } +}; + +Phi4GgufPackage::Phi4GgufPackage(std::unique_ptr impl) + : impl_(std::move(impl)) {} +Phi4GgufPackage::~Phi4GgufPackage() = default; + +std::shared_ptr Phi4GgufPackage::Open( + const std::filesystem::path& gguf_path) { + auto impl = std::make_unique(); + impl->path = gguf_path; + flm::file_access::ObserveOpen(gguf_path); + impl->file = CreateFileW(gguf_path.c_str(), GENERIC_READ, FILE_SHARE_READ, + nullptr, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr); + if (impl->file == INVALID_HANDLE_VALUE) + throw std::runtime_error("GGUF file: actual open failure " + + std::to_string(GetLastError()) + ", expected readable file"); + LARGE_INTEGER size; + if (!GetFileSizeEx(impl->file, &size) || size.QuadPart <= 0 || + static_cast(size.QuadPart) > std::numeric_limits::max()) + Fail("GGUF file size", std::to_string(size.QuadPart), "positive mappable size"); + impl->size = static_cast(size.QuadPart); + impl->mapping = CreateFileMappingW(impl->file, nullptr, PAGE_READONLY, 0, 0, nullptr); + if (!impl->mapping) + throw std::runtime_error("GGUF mapping: actual CreateFileMappingW failure " + + std::to_string(GetLastError()) + ", expected PAGE_READONLY mapping"); + impl->data = static_cast( + MapViewOfFile(impl->mapping, FILE_MAP_READ, 0, 0, 0)); + if (!impl->data) + throw std::runtime_error("GGUF mapping: actual MapViewOfFile failure " + + std::to_string(GetLastError()) + ", expected FILE_MAP_READ view"); + + const auto file = impl->bytes(); + Cursor cursor(file); + if (cursor.Read("GGUF header") != kMagic) + Fail("GGUF magic", "mismatch", "0x46554747"); + const auto version = cursor.Read("GGUF header"); + if (version != kVersion) Fail("GGUF version", std::to_string(version), "3"); + const auto tensor_count = cursor.Read("tensor count"); + const auto metadata_count = cursor.Read("metadata count"); + if (tensor_count > file.size() / 24) Fail("tensor count", std::to_string(tensor_count), "count fitting directory"); + if (metadata_count > file.size() / 12) Fail("metadata count", std::to_string(metadata_count), "count fitting metadata"); + + for (std::uint64_t i = 0; i < metadata_count; ++i) { + const auto key = cursor.ReadString("metadata key string"); + const auto type = cursor.Read(key); + const bool retain = IsRetainedKey(key); + auto value = ReadMetadataValue(cursor, type, key, retain); + if (retain) { + if (!impl->metadata.emplace(key, std::move(value)).second) + Fail(key, "duplicate metadata key", "unique metadata key"); + impl->metadata_types.emplace(key, type); + } + } + + constexpr std::uint64_t kDefaultAlignment = 32; + const auto alignment = impl->metadata.contains("general.alignment") + ? impl->Unsigned("general.alignment") + : kDefaultAlignment; + if (alignment == 0 || (alignment & (alignment - 1)) != 0) + Fail("general.alignment", std::to_string(alignment), "a non-zero power of two"); + + struct DirectoryTensor { + std::string name; + std::vector shape; + std::uint32_t type; + std::uint64_t relative_offset; + std::uint64_t length; + }; + std::vector directory; + directory.reserve(static_cast(tensor_count)); + for (std::uint64_t i = 0; i < tensor_count; ++i) { + auto name = cursor.ReadString("tensor directory name"); + const auto dimension_count = cursor.Read("tensor directory dimensions"); + if (dimension_count == 0 || dimension_count > 4) + Fail(name, std::to_string(dimension_count), "1..4 tensor dimensions"); + std::vector shape; + shape.reserve(dimension_count); + for (std::uint32_t d = 0; d < dimension_count; ++d) { + const auto dimension = cursor.Read("tensor directory dimension"); + if (dimension > static_cast(std::numeric_limits::max())) + Fail(name, std::to_string(dimension), "dimension fitting int64"); + shape.push_back(static_cast(dimension)); + } + std::reverse(shape.begin(), shape.end()); + const auto type = cursor.Read("tensor directory type"); + const auto offset = cursor.Read("tensor directory offset"); + const auto length = TensorByteLength(type, shape, name); + directory.push_back({std::move(name), std::move(shape), type, offset, length}); + } + + const auto data_start = AlignUp(cursor.offset(), alignment); + struct Range { std::uint64_t begin, end; std::string name; }; + std::vector ranges; + ranges.reserve(directory.size()); + for (auto& tensor : directory) { + if (tensor.relative_offset % alignment != 0) + Fail(tensor.name, std::to_string(tensor.relative_offset), "offset aligned to " + std::to_string(alignment)); + const auto absolute = CheckedAdd(data_start, tensor.relative_offset, tensor.name); + const auto bytes = RequireRange(file, absolute, tensor.length, tensor.name + " range"); + const auto end = CheckedAdd(absolute, tensor.length, tensor.name); + ranges.push_back({absolute, end, tensor.name}); + auto [it, inserted] = impl->tensors.emplace(tensor.name, + Impl::TensorRecord{tensor.name, bytes, std::move(tensor.shape), tensor.type, absolute}); + if (!inserted) Fail(tensor.name, "duplicate tensor name", "unique tensor name"); + } + std::sort(ranges.begin(), ranges.end(), [](const Range& a, const Range& b) { + return a.begin < b.begin; + }); + for (std::size_t i = 1; i < ranges.size(); ++i) { + if (ranges[i].begin < ranges[i - 1].end) + Fail(ranges[i].name, "overlap with " + ranges[i - 1].name, "non-overlapping tensor range"); + } + return std::shared_ptr(new Phi4GgufPackage(std::move(impl))); +} + +TensorView Phi4GgufPackage::RequireQ8( + std::string_view name, std::span expected_shape) const { + const auto& tensor = impl_->Tensor(name); + if (tensor.type != kTypeQ8_0) + Fail(name, GgmlTypeName(tensor.type), "Q8_0"); + if (!std::equal(tensor.shape.begin(), tensor.shape.end(), expected_shape.begin(), expected_shape.end())) + Fail(name, ShapeText(tensor.shape), ShapeText(expected_shape)); + const auto expected_length = TensorByteLength(kTypeQ8_0, expected_shape, name); + if (tensor.bytes.size() != expected_length) + Fail(name, std::to_string(tensor.bytes.size()) + " bytes", std::to_string(expected_length) + " bytes"); + return {tensor.name, tensor.bytes, tensor.shape, tensor.type}; +} + +FloatTensorView Phi4GgufPackage::RequireF32( + std::string_view name, std::span expected_shape) const { + const auto& tensor = impl_->Tensor(name); + if (tensor.type != kTypeF32) + Fail(name, GgmlTypeName(tensor.type), "F32"); + if (!std::equal(tensor.shape.begin(), tensor.shape.end(), expected_shape.begin(), expected_shape.end())) + Fail(name, ShapeText(tensor.shape), ShapeText(expected_shape)); + const auto expected_length = TensorByteLength(kTypeF32, expected_shape, name); + if (tensor.bytes.size() != expected_length) + Fail(name, std::to_string(tensor.bytes.size()) + " bytes", std::to_string(expected_length) + " bytes"); + const auto address = reinterpret_cast(tensor.bytes.data()); + if (tensor.absolute_offset % alignof(float) != 0 || address % alignof(float) != 0) + Fail(name, "address/offset not aligned", "alignment 4"); + return {tensor.name, + {reinterpret_cast(tensor.bytes.data()), + tensor.bytes.size() / sizeof(float)}, + tensor.shape}; +} + +ProjectionViews Phi4GgufPackage::AttentionQkv(std::size_t layer) const { + if (layer >= static_cast(kLayerCount)) + Fail("attention layer", std::to_string(layer), "0..31"); + const auto name = "blk." + std::to_string(layer) + ".attn_qkv.weight"; + const auto& tensor = impl_->Tensor(name); + if (tensor.shape.size() == 2 && tensor.shape[1] % 32 != 0) + Fail(name, std::to_string(tensor.shape[1]), "Q8_0 whole-row width divisible by 32"); + const auto fused = RequireQ8(name, std::array{5120, 3072}); + const auto input_width = fused.logical_shape[1]; + const auto row_bytes = static_cast(input_width / 32 * 34); + ProjectionViews result{}; + result.count = 3; + result.values[0] = {fused.name, fused.bytes.subspan(0, 3072 * row_bytes), {3072, 3072}, kTypeQ8_0}; + result.values[1] = {fused.name, fused.bytes.subspan(3072 * row_bytes, 1024 * row_bytes), {1024, 3072}, kTypeQ8_0}; + result.values[2] = {fused.name, fused.bytes.subspan(4096 * row_bytes, 1024 * row_bytes), {1024, 3072}, kTypeQ8_0}; + return result; +} + +ProjectionViews Phi4GgufPackage::GateUp(std::size_t layer) const { + if (layer >= static_cast(kLayerCount)) + Fail("MLP layer", std::to_string(layer), "0..31"); + const auto name = "blk." + std::to_string(layer) + ".ffn_up.weight"; + const auto& tensor = impl_->Tensor(name); + if (tensor.shape.size() == 2 && tensor.shape[1] % 32 != 0) + Fail(name, std::to_string(tensor.shape[1]), "Q8_0 whole-row width divisible by 32"); + const auto fused = RequireQ8(name, std::array{16384, 3072}); + const auto input_width = fused.logical_shape[1]; + const auto row_bytes = static_cast(input_width / 32 * 34); + ProjectionViews result{}; + result.count = 2; + result.values[0] = {fused.name, fused.bytes.subspan(0, 8192 * row_bytes), {8192, 3072}, kTypeQ8_0}; + result.values[1] = {fused.name, fused.bytes.subspan(8192 * row_bytes, 8192 * row_bytes), {8192, 3072}, kTypeQ8_0}; + return result; +} + +const std::filesystem::path& Phi4GgufPackage::Path() const { return impl_->path; } + +GgufPhi4Metadata Phi4GgufPackage::Metadata() const { + return {impl_->String("general.architecture"), + impl_->Unsigned("phi3.block_count"), + impl_->Unsigned("phi3.embedding_length"), + impl_->Unsigned("phi3.feed_forward_length"), + impl_->Unsigned("phi3.attention.head_count"), + impl_->Unsigned("phi3.attention.head_count_kv"), + impl_->Unsigned("phi3.context_length"), + impl_->Unsigned("phi3.rope.dimension_count"), + impl_->Number("phi3.rope.freq_base"), + impl_->Number("phi3.rope.scaling.attn_factor"), + impl_->Unsigned("phi3.rope.scaling.original_context_length"), + impl_->ArrayCount("tokenizer.ggml.tokens"), + impl_->Boolean("tokenizer.ggml.add_bos_token")}; +} + +void Phi4GgufPackage::ValidatePhi4Contract( + const nlohmann::json& config, const nlohmann::json& tokenizer, + const nlohmann::json& tokenizer_config) const { + const auto metadata = Metadata(); + const auto require_unsigned = [](std::string_view field, std::uint64_t actual, + std::uint64_t expected) { + if (actual != expected) Fail(field, std::to_string(actual), std::to_string(expected)); + }; + if (metadata.architecture != "phi3") Fail("general.architecture", metadata.architecture, "phi3"); + require_unsigned("phi3.block_count", metadata.layer_count, kLayerCount); + require_unsigned("phi3.context_length", metadata.context_length, kModelContextLength); + require_unsigned("phi3.embedding_length", metadata.hidden_size, kHiddenSize); + require_unsigned("phi3.feed_forward_length", metadata.intermediate_size, kIntermediateSize); + require_unsigned("phi3.attention.head_count", metadata.attention_head_count, kQueryHeadCount); + require_unsigned("phi3.attention.head_count_kv", metadata.kv_head_count, kKvHeadCount); + require_unsigned("phi3.rope.dimension_count", metadata.rope_dimension_count, kRopeDimension); + require_unsigned("phi3.rope.scaling.original_context_length", metadata.rope_original_context_length, kMaxSequenceLength); + require_unsigned("tokenizer.ggml.tokens", metadata.tokenizer_vocabulary_size, kVocabularySize); + if (metadata.add_bos_token) Fail("tokenizer.ggml.add_bos_token", "true", "false"); + const auto rms = impl_->Number("phi3.attention.layer_norm_rms_epsilon"); + if (!std::isfinite(rms) || rms != static_cast(kRmsEpsilon)) + Fail("phi3.attention.layer_norm_rms_epsilon", std::to_string(rms), std::to_string(kRmsEpsilon)); + for (const auto [field, value] : std::array{ + std::pair{"phi3.rope.freq_base", metadata.rope_frequency_base}, + std::pair{"phi3.rope.scaling.attn_factor", metadata.rope_attention_factor}}) { + if (!std::isfinite(value) || value <= 0) Fail(field, std::to_string(value), "finite positive value"); + } + + RequireQ8("token_embd.weight", std::array{kVocabularySize, kHiddenSize}); + RequireF32("output_norm.weight", std::array{kHiddenSize}); + if (impl_->tensors.contains("output.weight")) Fail("output.weight", "present", "absent (tied token_embd.weight)"); + if (impl_->tensors.contains("rope_factors_long.weight")) + RequireF32("rope_factors_long.weight", std::array{48}); + for (std::size_t layer = 0; layer < static_cast(kLayerCount); ++layer) { + const auto prefix = "blk." + std::to_string(layer); + RequireF32(prefix + ".attn_norm.weight", std::array{kHiddenSize}); + RequireF32(prefix + ".ffn_norm.weight", std::array{kHiddenSize}); + RequireQ8(prefix + ".attn_qkv.weight", std::array{kQueryDimension + 2 * kKvDimension, kHiddenSize}); + RequireQ8(prefix + ".attn_output.weight", std::array{kHiddenSize, kHiddenSize}); + RequireQ8(prefix + ".ffn_up.weight", std::array{2 * kIntermediateSize, kHiddenSize}); + RequireQ8(prefix + ".ffn_down.weight", std::array{kHiddenSize, kIntermediateSize}); + } + if (impl_->tensors.contains("rope_factors_short.weight")) + RequireF32("rope_factors_short.weight", std::array{48}); + + RequireJsonString(config, "model_type", "phi3"); + RequireJsonUnsigned(config, "num_hidden_layers", kLayerCount); + RequireJsonUnsigned(config, "hidden_size", kHiddenSize); + RequireJsonUnsigned(config, "intermediate_size", kIntermediateSize); + RequireJsonUnsigned(config, "num_attention_heads", kQueryHeadCount); + RequireJsonUnsigned(config, "num_key_value_heads", kKvHeadCount); + if (config.contains("head_dim")) + RequireJsonUnsigned(config, "head_dim", kHeadSize); + RequireJsonUnsigned(config, "vocab_size", kVocabularySize); + RequireJsonDouble(config, "rms_norm_eps", 1.0e-5); + RequireJsonUnsigned(config, "original_max_position_embeddings", kMaxSequenceLength); + RequireJsonUnsigned(config, "eos_token_id", 199999); + + std::set vocabulary_ids; + std::map> token_ids; + const auto add_token = [&](const std::string& token, const nlohmann::json& encoded_id) { + const std::string field = "tokenizer.json token ID " + token; + std::uint64_t unsigned_id; + if (encoded_id.is_number_unsigned()) { + unsigned_id = encoded_id.get(); + } else if (encoded_id.is_number_integer()) { + const auto signed_id = encoded_id.get(); + if (signed_id < 0) + Fail(field, std::to_string(signed_id), "0..200063"); + unsigned_id = static_cast(signed_id); + } else { + Fail(field, JsonText(encoded_id), "integer in 0..200063"); + } + if (unsigned_id >= static_cast(kVocabularySize)) + Fail(field, std::to_string(unsigned_id), "0..200063"); + const auto id = static_cast(unsigned_id); + const auto [it, inserted] = token_ids.emplace(token, id); + if (!inserted && it->second != id) + Fail(token, std::to_string(id), std::to_string(it->second)); + vocabulary_ids.insert(id); + }; + try { + const auto& vocab = tokenizer.at("model").at("vocab"); + if (!vocab.is_object()) Fail("tokenizer.json model.vocab", JsonText(vocab), "object mapping tokens to IDs"); + for (auto it = vocab.begin(); it != vocab.end(); ++it) + add_token(it.key(), it.value()); + const auto added = tokenizer.find("added_tokens"); + if (added != tokenizer.end()) { + if (!added->is_array()) Fail("tokenizer.json added_tokens", JsonText(*added), "array"); + for (const auto& item : *added) { + const auto content = item.at("content").get(); + add_token(content, item.at("id")); + } + } + } catch (const nlohmann::json::exception& error) { + Fail("tokenizer.json vocabulary", error.what(), "valid token-to-ID mappings"); + } + for (const auto& [token, expected] : std::array{ + std::pair{"<|end|>", 200020}, + std::pair{"<|endoftext|>", 199999}}) { + const auto it = token_ids.find(token); + if (it == token_ids.end()) Fail(token, "missing", std::to_string(expected)); + if (it->second != expected) Fail(token, std::to_string(it->second), std::to_string(expected)); + } + constexpr std::int64_t kTokenizerMaximumAssignedId = 200028; + constexpr std::size_t kTokenizerDistinctAssignedIds = 200029; + const auto actual_count = vocabulary_ids.size(); + const auto actual_max = vocabulary_ids.empty() ? -1 : *vocabulary_ids.rbegin(); + if (actual_max != kTokenizerMaximumAssignedId) + Fail("tokenizer.json maximum vocabulary ID", std::to_string(actual_max), + std::to_string(kTokenizerMaximumAssignedId)); + if (actual_count != kTokenizerDistinctAssignedIds) + Fail("tokenizer.json distinct vocabulary ID count", std::to_string(actual_count), + std::to_string(kTokenizerDistinctAssignedIds)); + const auto gguf_eos = impl_->Unsigned("tokenizer.ggml.eos_token_id"); + if (gguf_eos != 200020) Fail("tokenizer.ggml.eos_token_id", std::to_string(gguf_eos), "200020"); + + RequireJsonBoolean(tokenizer_config, "add_bos_token", false); + const auto template_it = tokenizer_config.find("chat_template"); + if (template_it == tokenizer_config.end() || !template_it->is_string()) + Fail("chat_template", template_it == tokenizer_config.end() ? "missing" : JsonText(*template_it), "string containing Phi-4 markers"); + const auto chat_template = template_it->get(); + const bool has_dynamic_role = + chat_template.find("'<|' + message['role'] + '|>'") != std::string::npos; + if (chat_template.find("<|user|>") == std::string::npos && !has_dynamic_role) + Fail("<|user|>", "missing from chat_template", "present in chat_template"); + for (const auto marker : {"<|end|>", "<|assistant|>"}) + if (chat_template.find(marker) == std::string::npos) + Fail(marker, "missing from chat_template", "present in chat_template"); +} + +} // namespace flm::phi4 diff --git a/src/common/models/phi4/rai/phi4_rai_host.cpp b/src/common/models/phi4/rai/phi4_rai_host.cpp new file mode 100644 index 000000000..1c3abbb89 --- /dev/null +++ b/src/common/models/phi4/rai/phi4_rai_host.cpp @@ -0,0 +1,177 @@ +#include "models/phi4/rai/phi4_rai_host.hpp" + +#include "models/phi4/rai/phi4_rai_constants.hpp" + +#include +#include +#include +#include +#include +#include +#include + +namespace flm::phi4 { +namespace { + +float HalfToFloat(std::uint16_t half) { + const std::uint32_t sign = static_cast(half & 0x8000) << 16; + const std::uint32_t exponent = (half >> 10) & 0x1f; + std::uint32_t fraction = half & 0x03ff; + std::uint32_t bits; + if (exponent == 0) { + if (fraction == 0) { + bits = sign; + } else { + int shift = 0; + while ((fraction & 0x0400) == 0) { + fraction <<= 1; + ++shift; + } + fraction &= 0x03ff; + bits = sign | (static_cast(127 - 14 - shift) << 23) | + (fraction << 13); + } + } else if (exponent == 0x1f) { + bits = sign | 0x7f800000 | (fraction << 13); + } else { + bits = sign | ((exponent + (127 - 15)) << 23) | (fraction << 13); + } + return std::bit_cast(bits); +} + +} // namespace + +std::vector DecodeEmbeddingRowsQ8( + const TensorView& embedding, std::span token_ids) { + if (embedding.ggml_type != 8 || embedding.logical_shape.size() != 2 || + embedding.logical_shape[0] <= 0 || embedding.logical_shape[1] <= 0 || + embedding.logical_shape[1] % 32 != 0) { + throw std::runtime_error("embedding must be a two-dimensional Q8_0 tensor with block-aligned rows"); + } + const auto rows = static_cast(embedding.logical_shape[0]); + const auto width = static_cast(embedding.logical_shape[1]); + const auto blocks_per_row = width / 32; + const auto row_bytes = blocks_per_row * 34; + if (rows > std::numeric_limits::max() / row_bytes || + embedding.bytes.size() != rows * row_bytes) { + throw std::runtime_error("embedding Q8_0 byte length does not match its logical shape"); + } + + std::vector result; + result.reserve(token_ids.size() * width); + for (const int token_id : token_ids) { + if (token_id < 0 || static_cast(token_id) >= rows) { + throw std::out_of_range("embedding token id is outside the vocabulary"); + } + const std::byte* row = embedding.bytes.data() + + static_cast(token_id) * row_bytes; + for (std::size_t block = 0; block < blocks_per_row; ++block) { + const std::byte* encoded = row + block * 34; + std::uint16_t scale_bits; + std::memcpy(&scale_bits, encoded, sizeof(scale_bits)); + const float scale = HalfToFloat(scale_bits); + for (std::size_t element = 0; element < 32; ++element) { + const auto code = static_cast( + std::to_integer(encoded[2 + element])); + result.push_back(scale * static_cast(code)); + } + } + } + return result; +} + +void HostRmsNorm( + std::span input, + std::span scale, + std::int64_t rows, + std::int64_t width, + float epsilon, + std::span output) { + if (rows <= 0 || width <= 0) + throw std::invalid_argument("Phi-4 RMSNorm rows and width must be positive"); + const auto row_count = static_cast(rows); + const auto row_width = static_cast(width); + if (row_count > std::numeric_limits::max() / row_width) + throw std::invalid_argument("Phi-4 RMSNorm shape overflow"); + const auto elements = row_count * row_width; + if (input.size() != elements || output.size() != elements || + scale.size() != row_width) + throw std::invalid_argument("Phi-4 RMSNorm shape mismatch"); + if (!std::isfinite(epsilon) || epsilon < 0.0f) + throw std::invalid_argument("Phi-4 RMSNorm epsilon must be finite and nonnegative"); + + for (std::size_t row = 0; row < row_count; ++row) { + const auto base = row * row_width; + double sum_of_squares = 0.0; + for (std::size_t column = 0; column < row_width; ++column) { + const double value = input[base + column]; + sum_of_squares += value * value; + } + const float mean_square = static_cast( + sum_of_squares / static_cast(width)); + const float denominator = std::sqrt(mean_square + epsilon); + for (std::size_t column = 0; column < row_width; ++column) + output[base + column] = + (input[base + column] / denominator) * scale[column]; + } +} + +std::vector ConvertF32ToBf16(std::span values) { + std::vector result; + result.reserve(values.size()); + for (const float value : values) { + std::uint32_t bits = std::bit_cast(value); + if ((bits & 0x7fffffffU) > 0x7f800000U) { + bits |= 0x00400000U; + } else { + bits += 0x7fffU + ((bits >> 16) & 1U); + } + result.push_back(static_cast(bits >> 16)); + } + return result; +} + +RopeTables BuildShortRopeTables( + const GgufPhi4Metadata& metadata, + std::optional short_factors) { + if (metadata.context_length < static_cast(kMaxSequenceLength) || + metadata.rope_original_context_length != static_cast(kMaxSequenceLength) || + metadata.rope_dimension_count != static_cast(kRopeDimension) || + !std::isfinite(metadata.rope_frequency_base) || metadata.rope_frequency_base <= 0 || + !std::isfinite(metadata.rope_attention_factor)) { + throw std::runtime_error("invalid Phi-4 RoPE metadata"); + } + + std::array factors{}; + factors.fill(1.0); + if (short_factors) { + if (short_factors->logical_shape != std::vector{kRopeDimension / 2} || + short_factors->values.size() != factors.size()) { + throw std::runtime_error("rope_factors_short.weight must have shape [48]"); + } + for (std::size_t i = 0; i < factors.size(); ++i) { + factors[i] = short_factors->values[i]; + if (!std::isfinite(factors[i]) || factors[i] <= 0) + throw std::runtime_error("rope_factors_short.weight must contain finite positive values"); + } + } + + RopeTables tables; + tables.cosine.resize(kMaxSequenceLength * factors.size()); + tables.sine.resize(kMaxSequenceLength * factors.size()); + for (std::size_t i = 0; i < factors.size(); ++i) { + const double inv_freq = 1.0 / + (std::pow(metadata.rope_frequency_base, (2.0 * i) / 96.0) * factors[i]); + for (std::size_t position = 0; position < kMaxSequenceLength; ++position) { + const double angle = static_cast(position) * inv_freq; + const auto index = position * factors.size() + i; + tables.cosine[index] = static_cast( + std::cos(angle) * metadata.rope_attention_factor); + tables.sine[index] = static_cast( + std::sin(angle) * metadata.rope_attention_factor); + } + } + return tables; +} + +} // namespace flm::phi4 diff --git a/src/common/models/phi4/rai/phi4_rai_shape_plan.cpp b/src/common/models/phi4/rai/phi4_rai_shape_plan.cpp new file mode 100644 index 000000000..529322e98 --- /dev/null +++ b/src/common/models/phi4/rai/phi4_rai_shape_plan.cpp @@ -0,0 +1,117 @@ +#include "models/phi4/rai/phi4_rai_shape_plan.hpp" + +#include "models/phi4/rai/phi4_rai_constants.hpp" + +#include +#include +#include +#include + +namespace flm::phi4 { +namespace { + +std::int64_t MatmulRows(const std::shared_ptr& api, + ryzenai_corelib_stream_ptr stream, + std::int64_t rows, std::int64_t logical_k, + std::int64_t logical_n, const char* logical_name) { + auto m = rows; + auto k = logical_k; + auto n = logical_n; + const std::string call = std::string("ryzenai_corelib_matmul_bf16_pad_shape ") + + logical_name + " [" + std::to_string(rows) + "," + + std::to_string(logical_k) + "]x[" + std::to_string(logical_k) + "," + + std::to_string(logical_n) + "]"; + api->Check(api->functions().matmul_pad_shape( + stream, &m, &k, &n, kRequantizedGroupSize), call); + if (k != logical_k || n != logical_n) { + throw std::runtime_error(call + ": helper changed padded K/N"); + } + return m; +} + +} // namespace + +Phi4ShapePlan Phi4ShapePlan::Build( + const std::shared_ptr& api, + ryzenai_corelib_stream_ptr stream) { + if (!api) throw std::invalid_argument("Phi4ShapePlan corelib API is null"); + if (!stream) throw std::invalid_argument("Phi4ShapePlan stream is null"); + + Phi4ShapePlan plan; + plan.attention_desc_ = {kQueryHeadCount, kKvHeadCount, kHeadSize, + kMaxSequenceLength, kRopeDimension}; + plan.lm_head_desc_ = {kHiddenSize, kVocabularySize, + kRequantizedGroupSize, false}; + plan.rows_.reserve(kMaxSequenceLength); + constexpr std::array execution_rows{ + 1, 64, 128, 256, 512, 1024, 2048, 4096}; + + for (const auto rows : execution_rows) { + Phi4RowExtents extents{}; + extents.query_rows = MatmulRows(api, stream, rows, kHiddenSize, + kQueryDimension, "query"); + extents.kv_rows = MatmulRows(api, stream, rows, kHiddenSize, + kKvDimension, "key/value"); + extents.output_rows = MatmulRows(api, stream, rows, kHiddenSize, + kHiddenSize, "output"); + + // 0.5.0 takes the whole weights descriptor here rather than k/n/group + // separately, because the activation and the post-feedforward norm + // select a different ELF family and therefore a different padding. + // Phi-4 is silu with no post-feedforward norm: both trailing fields 0. + const ryzenai_corelib_ssmlp_bf16_weights_desc ssmlp_desc{ + kHiddenSize, kIntermediateSize, kRequantizedGroupSize, 0, 0}; + extents.ssmlp_rows = rows; + const std::string ssmlp_call = + "ryzenai_corelib_ssmlp_bf16_pad_rows [" + std::to_string(rows) + + ",3072,8192]"; + api->Check(api->functions().ssmlp_pad_rows( + stream, &extents.ssmlp_rows, &ssmlp_desc), ssmlp_call); + + extents.flat_mha_rows = rows; + const std::string mha_call = + "ryzenai_corelib_flat_mha_bf16_pad_rows [" + std::to_string(rows) + + ",24,8,128,4096,96]"; + api->Check(api->functions().flat_mha_pad_rows( + stream, &extents.flat_mha_rows, &plan.attention_desc_), + mha_call); + plan.maximum_extents_.query_rows = std::max( + plan.maximum_extents_.query_rows, extents.query_rows); + plan.maximum_extents_.kv_rows = std::max( + plan.maximum_extents_.kv_rows, extents.kv_rows); + plan.maximum_extents_.output_rows = std::max( + plan.maximum_extents_.output_rows, extents.output_rows); + plan.maximum_extents_.ssmlp_rows = std::max( + plan.maximum_extents_.ssmlp_rows, extents.ssmlp_rows); + plan.maximum_extents_.flat_mha_rows = std::max( + plan.maximum_extents_.flat_mha_rows, extents.flat_mha_rows); + while (plan.rows_.size() < static_cast(rows)) + plan.rows_.push_back(extents); + } + + (void)MatmulRows(api, stream, 1, kHiddenSize, kVocabularySize, "lm_head"); + return plan; +} + +const Phi4RowExtents& Phi4ShapePlan::ForRows(std::size_t live_rows) const { + if (live_rows == 0 || live_rows > rows_.size()) { + throw std::out_of_range("Phi-4 live rows must be in 1..4096"); + } + return rows_[live_rows - 1]; +} + +const Phi4RowExtents& Phi4ShapePlan::maximum_extents() const noexcept { + return maximum_extents_; +} + +const ryzenai_corelib_flat_mha_bf16_desc& +Phi4ShapePlan::attention_desc() const noexcept { + return attention_desc_; +} + +const ryzenai_corelib_matmul_bf16_weights_desc& +Phi4ShapePlan::lm_head_desc() const noexcept { + return lm_head_desc_; +} + +} // namespace flm::phi4 diff --git a/src/common/models/phi4/rai/phi4_rai_weight_cache.cpp b/src/common/models/phi4/rai/phi4_rai_weight_cache.cpp new file mode 100644 index 000000000..4b9f942ae --- /dev/null +++ b/src/common/models/phi4/rai/phi4_rai_weight_cache.cpp @@ -0,0 +1,173 @@ +/// \file phi4_rai_weight_cache.cpp +/// \brief On-disk cache of the packed weights +#include "models/phi4/rai/phi4_rai_weight_cache.hpp" + +#include + +#include +#include +#include + +namespace flm::phi4 { +namespace { + +constexpr const char* kIndexName = "phi4-rai-weights.json"; +constexpr const char* kDataName = "phi4-rai-weights.bin"; +/// bumped when the on-disk layout changes in a way older indexes cannot express +constexpr int kIndexFormat = 1; + +bool IsDisabled(std::string_view value) { + return value == "0" || value == "off" || value == "OFF" || value == "false"; +} + +} // namespace + +bool WeightCacheKey::operator==(const WeightCacheKey& other) const { + return gguf_size == other.gguf_size && + gguf_write_time == other.gguf_write_time && + corelib_major == other.corelib_major && + corelib_minor == other.corelib_minor && + corelib_patch == other.corelib_patch && + group_size == other.group_size && + weight_count == other.weight_count; +} + +std::optional WeightCacheDirectory( + const std::filesystem::path& model_path) { + const char* configured = std::getenv("FLM_RAI_WEIGHT_CACHE"); + if (configured && *configured) { + if (IsDisabled(configured)) return std::nullopt; + return std::filesystem::path(configured); + } + return model_path; +} + +std::filesystem::path WeightCacheDataPath(const std::filesystem::path& directory) { + return directory / kDataName; +} + +WeightCacheKey MakeWeightCacheKey(const std::filesystem::path& gguf_path, + std::uint32_t corelib_major, + std::uint32_t corelib_minor, + std::uint32_t corelib_patch, + std::uint32_t group_size, + std::uint64_t weight_count) { + WeightCacheKey key; + key.corelib_major = corelib_major; + key.corelib_minor = corelib_minor; + key.corelib_patch = corelib_patch; + key.group_size = group_size; + key.weight_count = weight_count; + std::error_code error; + const auto size = std::filesystem::file_size(gguf_path, error); + if (!error) key.gguf_size = size; + const auto written = std::filesystem::last_write_time(gguf_path, error); + if (!error) key.gguf_write_time = written.time_since_epoch().count(); + return key; +} + +std::uint64_t RemoveWeightCache(const std::filesystem::path& directory) { + std::uint64_t reclaimed = 0; + const std::string index(kIndexName); + const std::string data(kDataName); + for (const auto& name : {data, data + ".tmp", index, index + ".tmp"}) { + std::error_code error; + const auto path = directory / name; + const auto size = std::filesystem::file_size(path, error); + if (error) continue; + if (std::filesystem::remove(path, error) && !error) reclaimed += size; + } + return reclaimed; +} + +std::optional ReadWeightCacheIndex( + const std::filesystem::path& directory, const WeightCacheKey& expected) { + try { + const auto index_path = directory / kIndexName; + std::ifstream input(index_path, std::ios::binary); + if (!input) return std::nullopt; + const auto document = nlohmann::json::parse(input, nullptr, false); + if (document.is_discarded()) return std::nullopt; + if (document.value("format", 0) != kIndexFormat) return std::nullopt; + + WeightCacheIndex index; + index.key.gguf_size = document.value("gguf_size", std::uint64_t{0}); + index.key.gguf_write_time = document.value("gguf_write_time", std::int64_t{0}); + index.key.corelib_major = document.value("corelib_major", std::uint32_t{0}); + index.key.corelib_minor = document.value("corelib_minor", std::uint32_t{0}); + index.key.corelib_patch = document.value("corelib_patch", std::uint32_t{0}); + index.key.group_size = document.value("group_size", std::uint32_t{0}); + index.key.weight_count = document.value("weight_count", std::uint64_t{0}); + if (!(index.key == expected)) return std::nullopt; + + const auto spans = document.find("spans"); + if (spans == document.end() || !spans->is_array()) return std::nullopt; + if (spans->size() != expected.weight_count) return std::nullopt; + index.spans.reserve(spans->size()); + for (const auto& span : *spans) { + if (!span.is_object()) return std::nullopt; + CachedWeightSpan entry; + entry.offset = span.value("offset", std::uint64_t{0}); + entry.size = span.value("size", std::uint64_t{0}); + if (entry.size == 0) return std::nullopt; + index.spans.push_back(entry); + } + + // The data file has to be at least as long as the last span claims, or + // the index is describing a file that was truncated under it. + std::error_code error; + const auto data_size = + std::filesystem::file_size(WeightCacheDataPath(directory), error); + if (error) return std::nullopt; + for (const auto& span : index.spans) { + if (span.offset + span.size > data_size) return std::nullopt; + } + return index; + } catch (...) { + // A damaged cache is a miss, never a failed load. + return std::nullopt; + } +} + +bool WriteWeightCacheIndex(const std::filesystem::path& directory, + const WeightCacheKey& key, + const std::vector& spans) { + try { + nlohmann::json document; + document["format"] = kIndexFormat; + document["gguf_size"] = key.gguf_size; + document["gguf_write_time"] = key.gguf_write_time; + document["corelib_major"] = key.corelib_major; + document["corelib_minor"] = key.corelib_minor; + document["corelib_patch"] = key.corelib_patch; + document["group_size"] = key.group_size; + document["weight_count"] = key.weight_count; + auto array = nlohmann::json::array(); + for (const auto& span : spans) { + array.push_back({{"offset", span.offset}, {"size", span.size}}); + } + document["spans"] = std::move(array); + + // Write to a temporary and rename, so a reader never sees a half index + // pointing into a data file it does not describe. + const auto final_path = directory / kIndexName; + const auto temporary = directory / (std::string(kIndexName) + ".tmp"); + { + std::ofstream output(temporary, std::ios::binary | std::ios::trunc); + if (!output) return false; + output << document.dump(); + if (!output) return false; + } + std::error_code error; + std::filesystem::rename(temporary, final_path, error); + if (error) { + std::filesystem::remove(temporary, error); + return false; + } + return true; + } catch (...) { + return false; + } +} + +} // namespace flm::phi4 diff --git a/src/common/npu_platform.cpp b/src/common/npu_platform.cpp new file mode 100644 index 000000000..8082c09bb --- /dev/null +++ b/src/common/npu_platform.cpp @@ -0,0 +1,13 @@ +/// \file npu_platform.cpp +/// \brief the NPU generation this build targets (stx vs aie_next) +#include "utils/npu_platform.hpp" + +namespace utils { + +std::optional parse_platform(std::string_view text) { + if (text == platform_id(npu_platform::stx)) return npu_platform::stx; + if (text == platform_id(npu_platform::aie_next)) return npu_platform::aie_next; + return std::nullopt; +} + +} // namespace utils diff --git a/src/common/rai/corelib_api.cpp b/src/common/rai/corelib_api.cpp new file mode 100644 index 000000000..1b5733a0d --- /dev/null +++ b/src/common/rai/corelib_api.cpp @@ -0,0 +1,206 @@ +#include "rai/corelib_api.hpp" + +#include +#include +#include +#include +#include +#include + +#ifdef _WIN32 +#ifndef NOMINMAX +#define NOMINMAX +#endif +#include +#endif + +namespace flm::corelib { +namespace { +/// \brief the corelib version this build was compiled against +/// \note Taken from the vendor header's own macros rather than written out +/// again. corelib_api.hpp already refuses to compile against any other +/// version, so deriving the run-time check from the same macros is what +/// keeps the two from drifting -- they were separate constants once, and +/// bumping one without the other is a mismatch that only shows up on a +/// machine with the real DLL. +constexpr CorelibVersion kRequiredVersion{RYZENAI_CORELIB_VERSION_MAJOR, + RYZENAI_CORELIB_VERSION_MINOR, + RYZENAI_CORELIB_VERSION_PATCH}; + +std::string VersionText(CorelibVersion version) { + return std::to_string(version.major) + "." + std::to_string(version.minor) + + "." + std::to_string(version.patch); +} + +std::string ErrorText(std::string_view call, + std::string_view status, + std::string_view detail) { + std::string result(call); + result += " failed: "; + result += status; + if (!detail.empty()) { + result += ": "; + result += detail; + } + return result; +} + +bool HasDllExtension(const std::filesystem::path& path) { + std::string extension = path.extension().string(); + std::transform(extension.begin(), extension.end(), extension.begin(), + [](unsigned char value) { + return static_cast(std::tolower(value)); + }); + return extension == ".dll"; +} +} // namespace + +CorelibError::CorelibError(ryzenai_corelib_status status, + std::string call, + std::string detail, + std::string status_text) + : std::runtime_error(ErrorText(call, status_text, detail)), + status_(status), + call_(std::move(call)), + detail_(std::move(detail)) {} + +ryzenai_corelib_status CorelibError::status() const noexcept { return status_; } +const std::string& CorelibError::call() const noexcept { return call_; } +const std::string& CorelibError::detail() const noexcept { return detail_; } + +CorelibApi::CorelibApi(Resolver resolver, + std::filesystem::path loaded_library_path) + : resolver_(std::move(resolver)), + loaded_library_path_(std::move(loaded_library_path)) { + void* version_symbol = resolver_("ryzenai_corelib_get_version"); + if (!version_symbol) { + throw std::runtime_error("missing corelib symbol: ryzenai_corelib_get_version"); + } + functions_.get_version = + reinterpret_cast(version_symbol); + functions_.get_version(&runtime_version_.major, &runtime_version_.minor, + &runtime_version_.patch); + if (runtime_version_.major != kRequiredVersion.major || + runtime_version_.minor != kRequiredVersion.minor || + runtime_version_.patch != kRequiredVersion.patch) { + throw std::runtime_error("corelib ABI mismatch: runtime " + + VersionText(runtime_version_) + ", required " + + VersionText(kRequiredVersion)); + } + +#define FLM_RESOLVE_CORELIB_FUNCTION(member, symbol) \ + if constexpr (std::string_view(#symbol) != \ + std::string_view("ryzenai_corelib_get_version")) { \ + void* address = resolver_(#symbol); \ + if (!address) throw std::runtime_error("missing corelib symbol: " #symbol); \ + functions_.member = reinterpret_cast(address); \ + } + FLM_CORELIB_FUNCTIONS(FLM_RESOLVE_CORELIB_FUNCTION) +#undef FLM_RESOLVE_CORELIB_FUNCTION +} + +std::shared_ptr CorelibApi::ResolveForTest( + Resolver resolver, std::filesystem::path loaded_library_path) { + if (!resolver) throw std::invalid_argument("corelib resolver is empty"); + return std::shared_ptr(new CorelibApi( + std::move(resolver), std::move(loaded_library_path))); +} + +std::shared_ptr CorelibApi::Load(const std::filesystem::path& dll) { +#ifndef _WIN32 + (void)dll; + throw std::runtime_error("ryzenai-corelib loading currently requires Windows"); +#else + const std::filesystem::path absolute_dll = std::filesystem::absolute(dll); + HMODULE raw_module = LoadLibraryExW( + absolute_dll.c_str(), nullptr, + LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_DEFAULT_DIRS); + if (!raw_module) { + throw std::runtime_error("failed to load corelib DLL '" + + absolute_dll.string() + "' (Windows error " + + std::to_string(GetLastError()) + ")"); + } + auto module = std::shared_ptr(raw_module, [](void* handle) { + FreeLibrary(static_cast(handle)); + }); + Resolver resolver = [module](std::string_view name) -> void* { + const std::string terminated(name); + return reinterpret_cast( + GetProcAddress(static_cast(module.get()), terminated.c_str())); + }; + return ResolveForTest(std::move(resolver), absolute_dll); +#endif +} + +#if defined(FLM_CORELIB_LINK_STATIC) +std::shared_ptr CorelibApi::LoadStatic() { + const char* configured = std::getenv("FLM_RAI_CORELIB_PATH"); + if (configured && *configured) { + std::cerr << "[FLM] Ignoring FLM_RAI_CORELIB_PATH: corelib is linked " + "into this build, so there is no DLL to select." + << std::endl; + } + // Resolve through the same name-keyed resolver the DLL path uses, so the + // version check and the missing-symbol diagnostics stay identical. + Resolver resolver = [](std::string_view name) -> void* { +#define FLM_STATIC_CORELIB_SYMBOL(member, symbol) \ + if (name == std::string_view(#symbol)) { \ + return reinterpret_cast(&::symbol); \ + } + FLM_CORELIB_FUNCTIONS(FLM_STATIC_CORELIB_SYMBOL) +#undef FLM_STATIC_CORELIB_SYMBOL + return nullptr; + }; + return ResolveForTest(std::move(resolver), + std::filesystem::path("")); +} +#endif + +std::filesystem::path CorelibApi::ResolveLibraryPath( + const std::filesystem::path& executable_dir) { + const char* configured = std::getenv("FLM_RAI_CORELIB_PATH"); + if (configured && *configured) { + const std::filesystem::path path(configured); + if (!path.is_absolute()) { + throw std::runtime_error( + "FLM_RAI_CORELIB_PATH must be an absolute .dll path"); + } + if (!path.has_filename() || !HasDllExtension(path)) { + throw std::runtime_error( + "FLM_RAI_CORELIB_PATH must name an absolute .dll file"); + } + return path; + } + return std::filesystem::absolute(executable_dir / "rai" / + "ryzenai_corelib.dll"); +} + +const CorelibFunctions& CorelibApi::functions() const noexcept { return functions_; } +CorelibVersion CorelibApi::runtime_version() const noexcept { return runtime_version_; } +const std::filesystem::path& CorelibApi::loaded_library_path() const noexcept { + return loaded_library_path_; +} + +void CorelibApi::Check(ryzenai_corelib_status status, + std::string_view call) const { + if (status == ryzenai_corelib_status_success) return; + const char* detail_pointer = functions_.get_last_error_message(); + const std::string detail = detail_pointer ? detail_pointer : ""; + const char* status_pointer = functions_.status_to_string(status); + const std::string status_text = status_pointer ? status_pointer : "unknown"; + throw CorelibError(status, std::string(call), detail, status_text); +} + +void CorelibApi::RegisterObject() const noexcept { ++live_object_count_; } + +void CorelibApi::Release(void* object) const noexcept { + if (!object) return; + functions_.object_release(object); + --live_object_count_; +} + +std::size_t CorelibApi::live_object_count() const noexcept { + return live_object_count_.load(); +} + +} // namespace flm::corelib diff --git a/src/common/rai/corelib_runtime.cpp b/src/common/rai/corelib_runtime.cpp new file mode 100644 index 000000000..3e1d5ea21 --- /dev/null +++ b/src/common/rai/corelib_runtime.cpp @@ -0,0 +1,110 @@ +#include "rai/corelib_runtime.hpp" + +#include +#include + +namespace flm::corelib { +namespace { +std::mutex process_mutex; +std::shared_ptr process_runtime; +#if defined(FLM_CORELIB_TESTING) +std::function destruction_observer; +bool shutdown_execution_lock_held = false; +#endif +} + +CorelibRuntime::CorelibRuntime(std::shared_ptr api) + : api_(std::move(api)) {} + +CorelibRuntime::~CorelibRuntime() { +#if defined(FLM_CORELIB_TESTING) + if (destruction_observer) destruction_observer(shutdown_execution_lock_held); +#endif +} + +std::shared_ptr CorelibRuntime::CreateReady( + std::shared_ptr api) { + if (!api) throw std::invalid_argument("corelib API is null"); + api->Check(api->functions().selftest_dependencies(), + "ryzenai_corelib_selftest_dependencies"); + // 0.5.0 replaced has_device_context() with get_device(): the same question, + // answered by a pointer. NULL means no NPU to dispatch to -- packing and + // padding still work, everything that binds device memory does not. + if (api->functions().get_device() == nullptr) { + throw std::runtime_error("corelib has no device context"); + } + return std::shared_ptr(new CorelibRuntime(std::move(api))); +} + +std::shared_ptr CorelibRuntime::GetOrCreate( + const std::filesystem::path& executable_dir) { + std::lock_guard lock(process_mutex); + if (!process_runtime) { +#if defined(FLM_CORELIB_LINK_STATIC) + (void)executable_dir; + auto api = CorelibApi::LoadStatic(); +#else + auto api = CorelibApi::Load(CorelibApi::ResolveLibraryPath(executable_dir)); +#endif + process_runtime = CreateReady(std::move(api)); + } + return process_runtime; +} + +std::shared_ptr CorelibRuntime::CreateForTest( + std::shared_ptr api) { + auto runtime = CreateReady(std::move(api)); + std::lock_guard lock(process_mutex); + if (process_runtime) { + throw std::runtime_error("corelib runtime already exists"); + } + process_runtime = runtime; + return runtime; +} + +void CorelibRuntime::ShutdownProcess() { + std::lock_guard process_lock(process_mutex); + if (!process_runtime) return; + + auto runtime = process_runtime; + std::unique_lock execution_lock(runtime->execution_mutex_); +#if defined(FLM_CORELIB_TESTING) + shutdown_execution_lock_held = true; +#endif + if (runtime->api_->live_object_count() != 0) { +#if defined(FLM_CORELIB_TESTING) + shutdown_execution_lock_held = false; +#endif + throw std::runtime_error("cannot shut down with live corelib objects"); + } + runtime->api_->functions().cleanup(); + runtime->api_.reset(); + process_runtime.reset(); + execution_lock.unlock(); +#if defined(FLM_CORELIB_TESTING) + shutdown_execution_lock_held = false; +#endif + runtime.reset(); +} + +#if defined(FLM_CORELIB_TESTING) +void CorelibRuntime::SetDestructionObserverForTest( + std::function observer) { + destruction_observer = std::move(observer); +} +#endif + +std::unique_lock CorelibRuntime::AcquireExecution() { + return std::unique_lock(execution_mutex_); +} + +const std::shared_ptr& CorelibRuntime::api() const noexcept { + return api_; +} + +const std::filesystem::path& CorelibRuntime::loaded_library_path() const noexcept { + static const std::filesystem::path empty; + return api_ ? api_->loaded_library_path() : empty; +} + +} // namespace flm::corelib diff --git a/src/common/rai/rai_sources.cmake b/src/common/rai/rai_sources.cmake new file mode 100644 index 000000000..892c4e053 --- /dev/null +++ b/src/common/rai/rai_sources.cmake @@ -0,0 +1,6 @@ +include("${CMAKE_CURRENT_LIST_DIR}/../models/models_sources.cmake") + +set(FLM_RAI_SOURCES + "${CMAKE_CURRENT_LIST_DIR}/corelib_api.cpp" + "${CMAKE_CURRENT_LIST_DIR}/corelib_runtime.cpp" + ${FLM_MODELS_RAI_SOURCES}) diff --git a/src/common/tokenizer/tokenizer.cpp b/src/common/tokenizer/tokenizer.cpp index 2b21981cd..f048fc852 100644 --- a/src/common/tokenizer/tokenizer.cpp +++ b/src/common/tokenizer/tokenizer.cpp @@ -4,6 +4,7 @@ /// \date 2025-06-24 /// \version 0.9.10 #include "tokenizer/tokenizer.hpp" +#include "utils/file_access.hpp" #include #include #include @@ -15,6 +16,8 @@ /// \brief Constructor /// \param model_path the model path Tokenizer::Tokenizer(const std::string& model_path) { + flm::file_access::ObserveOpen( + std::filesystem::path(model_path) / "tokenizer.json"); #ifdef _WIN32 std::ifstream fs(model_path + "\\tokenizer.json", std::ios::in | std::ios::binary); #else diff --git a/src/include/AutoModel/automodel.hpp b/src/include/AutoModel/automodel.hpp index 45a6a63b7..5708b17f3 100644 --- a/src/include/AutoModel/automodel.hpp +++ b/src/include/AutoModel/automodel.hpp @@ -6,6 +6,7 @@ /// \note This is a header file for the auto_model class #pragma once +#include #include #include #include @@ -16,9 +17,12 @@ #include #include #include +#include +#include #include "typedef.hpp" #include "causal_lm.hpp" #include "lm_config.hpp" +#include "AutoModel/model_backend.hpp" #include "models/llama/llama_npu.hpp" #include "models/qwen2/qwen2_npu.hpp" #include "models/qwen3/qwen3_npu.hpp" @@ -142,17 +146,35 @@ struct lm_uniform_input_t { std::vector audios; std::vector audio_payload_types; nlohmann::ordered_json tools; + std::optional requested_max_new_tokens; }; +inline std::optional normalize_requested_max_new_tokens( + std::optional requested) { + return requested.has_value() && *requested > 0 ? requested : std::nullopt; +} + using json = nlohmann::ordered_json; +class ModelRequestError final : public std::runtime_error { +public: + ModelRequestError(int http_code, bool session_cleared, std::string message); + int http_code() const noexcept; + bool session_cleared() const noexcept; +private: + int http_code_; + bool session_cleared_; +}; + class AutoModel { protected: std::string model_path = ""; - std::unique_ptr lm_engine = nullptr; + /// \brief the execution backend, which owns the engine + std::unique_ptr backend_ = nullptr; + /// \brief a non-owning view of backend_'s engine, or null when none is loaded + causal_lm* lm_engine = nullptr; std::unique_ptr tokenizer = nullptr; std::unique_ptr sampler = nullptr; - std::unique_ptr q4nx = nullptr; bool is_model_loaded = false; std::string current_model = ""; std::vector token_history; @@ -215,14 +237,57 @@ class AutoModel { void _shared_load_model(std::string model_path, json model_info, int default_context_length = -1, bool enable_preemption = false); + void _shared_initialize_model_state(std::string model_path, json model_info, int context_length); + void _shared_initialize_legacy_npu(bool enable_preemption); nlohmann::json _shared_setup_tokenizer(std::string model_path); + /// \brief Load a model onto its chosen backend + /// \param model_path the model directory + /// \param model_info the resolved model_list.json entry + /// \param default_context_length the requested context length, or -1 for the catalog default + /// \param enable_preemption whether preemption was asked for + /// \param requested_backend the --backend value, empty when it was not given + /// \note Resolves the backend, initializes the shared state, builds the + /// backend through the registry and points lm_engine at its engine. + /// Every frontend calls this instead of doing it by hand; what is left + /// for the frontend is the tokenizer, the chat template and the sampler. + /// \throws std::runtime_error if the backend is unknown, unavailable for the + /// model, or cannot honour the requested preemption/context length + void _shared_load_backend(std::string model_path, json model_info, int default_context_length = -1, bool enable_preemption = false, const std::string& requested_backend = "", const nlohmann::json* tokenizer_config = nullptr); + + /// \brief Drop the conversation after a failed inference + /// \param poisoned whether the engine can no longer be driven at all + /// \note A poisoned engine is not asked to clear its context; only a reload + /// can recover it. + void _shared_after_inference_failure(bool poisoned); + + /// \brief Refuse to use an engine that needs a reload + /// \throws ModelRequestError 500 when the backend reports itself poisoned + void _shared_guard_poisoned() const; + + /// \brief The most tokens the loaded model may hold + /// \return MAX_L, lowered to the backend's own decode limit when it has one + uint32_t decode_cap() const { + const uint32_t backend_cap = + backend_ ? backend_->max_decode_length() : 0; + return backend_cap == 0 ? MAX_L : std::min(MAX_L, backend_cap); + } + /// \brief Insert tokens into the model /// \param meta_info the meta information of the chat /// \param tokens the tokens to insert /// \param payload the payload, it shall not be used as this function is only used for chunkwised insertion, no image allowed /// \return true if the tokens were inserted successfully, false otherwise - bool _shared_insert(chat_meta_info_t& meta_info, std::vector& tokens, std::function is_cancelled = [] { return false; }, void* payload = nullptr, int first_len_run = 0); + bool _shared_insert(chat_meta_info_t& meta_info, std::vector& tokens, std::function is_cancelled = [] { return false; }, void* payload = nullptr, int first_len_run = 0, std::optional requested_max_new_tokens = std::nullopt); + + /// \brief Reject a request that cannot fit in the backend's decode limit + /// \param rendered_tokens the length of the rendered prompt + /// \param requested the caller's max_new_tokens, if any + /// \note Only backends that declare a hard decode limit of their own are + /// checked; the FastFlowLM engines are bounded by MAX_L alone and keep + /// their existing behaviour of truncating rather than refusing. + /// \throws ModelRequestError 400 when prompt plus output cannot fit + void _shared_validate_capacity(std::size_t rendered_tokens, std::optional requested) const; buffer _chunked_insert(chat_meta_info_t& meta_info, std::vector& tokens, std::function is_cancelled = [] { return false; }, void* payload = nullptr, int first_len_run = 0); std::string _shared_generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }); @@ -263,6 +328,12 @@ class AutoModel { /// \return the current model std::string get_current_model(); + /// \brief Get the id of the backend the model is loaded on + /// \return the backend id, or empty when no model is loaded + virtual std::string backend_id() const noexcept { + return backend_ ? backend_->id() : std::string(); + } + /// \brief Get the current context length /// \return the current context length virtual int get_current_context_length(); @@ -407,7 +478,7 @@ class AutoModel { //************ Unique for each model *************/ - virtual void load_model(std::string model_path, json model_info, int default_context_length = -1, bool enable_preemption = false) {} + virtual void load_model(std::string model_path, json model_info, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") {} virtual std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) = 0; virtual chat_template_type_t get_chat_template_type() { return chat_template_type_t::chat_ml; @@ -431,6 +502,12 @@ class AutoModel { /// \brief Generate the tokens with prompt virtual std::string generate_with_prompt(chat_meta_info_t& meta_info, lm_uniform_input_t& input, int length_limit, std::ostream& os = std::cout) = 0; + std::string generate_with_prompt( + chat_meta_info_t& meta_info, + lm_uniform_input_t& input, + int length_limit, + std::ostream& os, + std::function is_cancelled); /// \brief Configure a parameter with type-erased value /// \param parameter_name the name of the parameter diff --git a/src/include/AutoModel/flm_backend.hpp b/src/include/AutoModel/flm_backend.hpp new file mode 100644 index 000000000..20e4fecab --- /dev/null +++ b/src/include/AutoModel/flm_backend.hpp @@ -0,0 +1,64 @@ +/// \file flm_backend.hpp +/// \brief The flm backend, shared by every family that ships one +/// \note The flm backend runs FastFlowLM's own NPU kernels, and every one of +/// those engines is built the same way -- construct, load the Q4NX +/// weights, clear the context -- so one template covers them all. +#pragma once + +#include "AutoModel/model_backend.hpp" +#include "tensor_utils/q4_npu_eXpress.hpp" + +#include +#include +#include +#include + +namespace flm::backend { + +/// \brief the flm backend for a FastFlowLM NPU engine +/// \tparam Engine the concrete engine type, e.g. phi4_npu +/// \note load_weights is called on Engine*, not on causal_lm*. The pure virtual +/// stays in the frozen causal_lm.hpp for ABI, but nothing here depends on +/// it, so a backend whose engine loads its weights some other way is under +/// no obligation to pretend otherwise. +template +class FlmBackend final : public ModelBackend { +public: + explicit FlmBackend(const BackendContext& context) { + if (!context.config) { + throw std::runtime_error("flm backend needs an LM_Config"); + } + if (!context.npu) { + throw std::runtime_error("flm backend needs an NPU instance"); + } + + // Scoped: the packed weights are copied into the engine, and the + // several hundred MB they occupy are freed before load_model returns. + Q4NX q4nx(context.model_path); + auto engine = std::make_unique( + *context.config, context.npu, + static_cast(context.context_length)); + engine->load_weights(q4nx); + engine->clear_context(); + engine_ = std::move(engine); + } + + causal_lm& engine() override { return *engine_; } + + std::string id() const override { return kFlmBackendId; } + +private: + std::unique_ptr engine_; +}; + +/// \brief a factory building FlmBackend +/// \tparam Engine the concrete engine type +/// \return a factory suitable for BackendRegistry::register_backend +template +BackendFactory flm_factory() { + return [](const BackendContext& context) -> std::unique_ptr { + return std::make_unique>(context); + }; +} + +} // namespace flm::backend diff --git a/src/include/AutoModel/model_backend.hpp b/src/include/AutoModel/model_backend.hpp new file mode 100644 index 000000000..3b55fdda4 --- /dev/null +++ b/src/include/AutoModel/model_backend.hpp @@ -0,0 +1,207 @@ +/// \file model_backend.hpp +/// \brief Named execution backends for a model family +/// \note A backend names *where the kernels come from*, not which silicon runs +/// them: `flm` is FastFlowLM's own kernel flow, `rai` reaches them +/// through ryzenai-corelib. Both are causal_lm subclasses, but they +/// differ in how they are built and in how they must be driven. A +/// ModelBackend owns one engine and states those differences, so the +/// frontends stay backend-agnostic. +/// \note A backend id is *not* a platform id. Which generation a build targets +/// is utils::npu_platform's job; the two axes line up one-to-one today, +/// which is exactly why they need separate names -- otherwise the day +/// they stop lining up is the day the strings start lying. +/// \note This seam deliberately sits *above* causal_lm. The engine libraries in +/// src/lib/ are prebuilt against causal_lm.hpp, so that header is +/// a frozen ABI: adding or reordering a virtual there would silently shift +/// vtable slots in code this build cannot recompile. +#pragma once + +#include "causal_lm.hpp" +#include "lm_config.hpp" +#include "device_runtime.hpp" +#include "nlohmann/json.hpp" + +#include +#include +#include +#include +#include +#include +#include + +class npu_xclbin_manager; + +namespace flm::backend { + +/// \brief FastFlowLM's own NPU kernel flow +/// \note Every model family registers this one; it is what a build without +/// FLM_ENABLE_RAI runs, on whatever generation it was built for. +inline constexpr const char* kFlmBackendId = "flm"; + +/// \brief kernels reached through ryzenai-corelib +/// \note Only compiled in when FLM_ENABLE_RAI is on, and only for families that +/// have a corelib engine. +inline constexpr const char* kRaiBackendId = "rai"; + +/// \brief the backend a build falls back to when nothing else picks one +/// \return kRaiBackendId when built with FLM_ENABLE_RAI, kFlmBackendId otherwise +/// \note A build links one kernel flow or the other, never both, so this is a +/// compile-time fact. It is deliberately *not* derived from +/// utils::platform_id: the backend axis and the platform axis are allowed +/// to disagree, and routing one through the other is what used to make +/// them look like the same thing. +constexpr const char* build_default_backend_id() { +#ifdef FLM_ENABLE_RAI + return kRaiBackendId; +#else + return kFlmBackendId; +#endif +} + +/// \brief everything a backend factory needs to build its engine +/// \note Assembled by the frontend once the shared model state is initialized, +/// so `config` and `context_length` are already resolved. +struct BackendContext { + std::string model_path; + nlohmann::ordered_json model_info; + const LM_Config* config = nullptr; + /// \note null for backends that do not drive the NPU through an xclbin + npu_xclbin_manager* npu = nullptr; + flm_rt::device* device = nullptr; + std::uint32_t context_length = 0; + bool enable_preemption = false; + /// \brief the frontend's parse of tokenizer_config.json + /// \note Supplied here rather than read by the backend, so the model + /// directory layout stays the frontend's knowledge and the file is + /// opened once per load. null for families that have no such file. + const nlohmann::json* tokenizer_config = nullptr; +}; + +/// \brief what a backend needs from the frontend *before* it can be built +/// \note These cannot live on ModelBackend: the frontend has to know them to +/// assemble the BackendContext in the first place. The defaults describe +/// the FastFlowLM NPU engines. +struct BackendTraits { + /// \brief whether the frontend should build an npu_xclbin_manager for it + bool needs_npu_xclbin = true; + /// \brief whether this backend can run with preemption enabled + bool supports_preemption = true; + /// \brief the largest context length it accepts, or 0 when it has no ceiling + std::uint32_t max_context_length = 0; +}; + +/// \brief one engine plus every rule for driving it +/// \note The defaults describe the FastFlowLM NPU engines, so a backend only +/// has to state what makes it different. +class ModelBackend { +public: + virtual ~ModelBackend() = default; + + /// \brief the engine this backend owns + virtual causal_lm& engine() = 0; + + /// \brief the registered id of this backend, e.g. "flm" + virtual std::string id() const = 0; + + /// \brief one line of provenance for `flm show`, empty when there is none + virtual std::string detail() const { return {}; } + + /// \brief hard ceiling on decoded tokens, or 0 when only MAX_L applies + virtual std::uint32_t max_decode_length() const { return 0; } + + /// \brief whether this backend can run with preemption enabled + virtual bool supports_preemption() const { return true; } + + /// \brief whether the engine wants one more forward() after an EOS token + /// \note The flm engines use it to keep their KV cache in step; the rai + /// engines reject a decode past corelib's own limit, so they opt out. + virtual bool forwards_past_eos() const { return true; } + + /// \brief whether the engine has failed in a way that needs a full reload + virtual bool poisoned() const noexcept { return false; } + + /// \brief EOS ids proven by the backend's own package, when it has them + /// \note Returning a value overrides whatever tokenizer_config.json says. + virtual std::optional> forced_eos_ids() const { + return std::nullopt; + } +}; + +using BackendFactory = + std::function(const BackendContext&)>; + +/// \brief the family -> (backend id -> factory) table +class BackendRegistry { +public: + /// \brief the process-wide registry, populated with the built-in backends + static BackendRegistry& instance(); + + /// \brief register a backend for a family + /// \throws std::runtime_error if that family already has that id + void register_backend(std::string family, std::string id, + BackendFactory factory, BackendTraits traits = {}); + + /// \brief register a backend, replacing any backend already under that id + /// \note Unlike register_backend this never throws on a duplicate. It exists + /// for tests, which swap a real engine for a stub; production code + /// registers once, through register_builtin_backends. + void replace_backend(std::string family, std::string id, + BackendFactory factory, BackendTraits traits = {}); + + /// \brief the ids registered for a family, sorted + std::vector available(const std::string& family) const; + + /// \brief what a registered backend needs before it can be built + /// \throws std::runtime_error naming the available ids if it is not registered + BackendTraits traits(const std::string& family, + const std::string& id) const; + + /// \brief whether a family has a backend with this id + bool has(const std::string& family, const std::string& id) const; + + /// \brief build a backend + /// \throws std::runtime_error naming the available ids if it is not registered + std::unique_ptr create(const std::string& family, + const std::string& id, + const BackendContext& context) const; + +private: + struct Entry { + BackendFactory factory; + BackendTraits traits; + }; + + /// \brief look up one entry, or throw naming what the family does provide + Entry lookup(const std::string& family, const std::string& id) const; + + mutable std::mutex mutex_; + std::map> factories_; +}; + +/// \brief register every backend that ships with this build +/// \param registry the registry to populate +/// \note Defined in builtin_backends.cpp, which is the one place that knows +/// both the family names and the engine types. +void register_builtin_backends(BackendRegistry& registry); + +/// \brief decide which backend to run a model on +/// \param family the model family, as in details.family +/// \param fallback the backend id to use when nothing else picks one, normally +/// build_default_backend_id() +/// \param requested the --backend value, empty when the flag was not given +/// \param source if non-null, receives a human-readable reason for the choice +/// \return the resolved backend id +/// \throws std::runtime_error naming the registered ids when nothing matches +/// \note Precedence: --backend, then FLM_BACKEND, then `fallback`. The catalog +/// does not name a backend: model_list has already pruned itself to the +/// entries this build's platform can run, and a per-entry backend list +/// would only restate what the build already links. +/// \note `fallback` arrives as a string so that this header stays free of the +/// NPU runtime includes, which is what lets test/model_backend build +/// without an XRT toolchain. +std::string resolve_backend_id(const std::string& family, + const std::string& fallback, + const std::string& requested = "", + std::string* source = nullptr); + +} // namespace flm::backend diff --git a/src/include/AutoModel/modeling_gemma3.hpp b/src/include/AutoModel/modeling_gemma3.hpp index e4dd88c92..0b539bc02 100644 --- a/src/include/AutoModel/modeling_gemma3.hpp +++ b/src/include/AutoModel/modeling_gemma3.hpp @@ -29,7 +29,7 @@ class Gemma3 : public AutoModel { public: Gemma3(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; diff --git a/src/include/AutoModel/modeling_gemma3_text.hpp b/src/include/AutoModel/modeling_gemma3_text.hpp index b9c5b70c1..a6cdee184 100644 --- a/src/include/AutoModel/modeling_gemma3_text.hpp +++ b/src/include/AutoModel/modeling_gemma3_text.hpp @@ -31,7 +31,7 @@ class Gemma3_Text_Only : public AutoModel { public: Gemma3_Text_Only(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; diff --git a/src/include/AutoModel/modeling_gemma4_12b.hpp b/src/include/AutoModel/modeling_gemma4_12b.hpp index 968fd4f14..ff971ec44 100644 --- a/src/include/AutoModel/modeling_gemma4_12b.hpp +++ b/src/include/AutoModel/modeling_gemma4_12b.hpp @@ -101,7 +101,7 @@ class Gemma4_12B : public AutoModel { public: Gemma4_12B(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; std::string generate_with_prompt(chat_meta_info_t& meta_info, lm_uniform_input_t& input, int length_limit, std::ostream& os = std::cout) override; diff --git a/src/include/AutoModel/modeling_gemma4e.hpp b/src/include/AutoModel/modeling_gemma4e.hpp index 45736a668..1a32378b8 100644 --- a/src/include/AutoModel/modeling_gemma4e.hpp +++ b/src/include/AutoModel/modeling_gemma4e.hpp @@ -111,15 +111,17 @@ class Gemma4e : public AutoModel { /// \note Overridden by Gemma4e_Flash to read them off the flash engine. virtual gemma4e_engine_config_t engine_config() const; - /// \brief Builds the engine behind this wrapper; load_model() calls it. - /// \note Overridden by Gemma4e_Flash to swap in the flash engine. Everything - /// else -- weights, tokenizer, sampler, chat template -- is identical. - virtual void create_engine(); + /// \note Gemma4e and Gemma4e_Flash share this whole wrapper -- weights, + /// tokenizer, sampler, chat template -- and differ only in the engine + /// behind it. Which one to build is no longer decided here: the two + /// are separate families in the catalog ("gemma4e" and + /// "gemma4e-flash") and each registers its own backend, so the + /// registry picks the engine. See AutoModel/builtin_backends.cpp. public: Gemma4e(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; std::string generate_with_prompt(chat_meta_info_t& meta_info, lm_uniform_input_t& input, int length_limit, std::ostream& os = std::cout) override; @@ -229,7 +231,6 @@ class Gemma4e_Flash : public Gemma4e { void _reset_turn(); protected: - void create_engine() override; gemma4e_engine_config_t engine_config() const override; public: diff --git a/src/include/AutoModel/modeling_gpt_oss.hpp b/src/include/AutoModel/modeling_gpt_oss.hpp index f04511997..ea6747771 100644 --- a/src/include/AutoModel/modeling_gpt_oss.hpp +++ b/src/include/AutoModel/modeling_gpt_oss.hpp @@ -35,7 +35,7 @@ class GPT_OSS : public AutoModel { public: GPT_OSS(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_info, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_info, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; std::string apply_chat_template(nlohmann::ordered_json& messages, nlohmann::ordered_json tools = nlohmann::ordered_json::object()) override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; diff --git a/src/include/AutoModel/modeling_hunyuan.hpp b/src/include/AutoModel/modeling_hunyuan.hpp index e033d4fba..e31b10d4d 100644 --- a/src/include/AutoModel/modeling_hunyuan.hpp +++ b/src/include/AutoModel/modeling_hunyuan.hpp @@ -49,7 +49,7 @@ class Hunyuan : public AutoModel { public: Hunyuan(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_info, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_info, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; std::string generate_with_prompt(chat_meta_info_t& meta_info, lm_uniform_input_t& input, int length_limit, std::ostream& os = std::cout) override; diff --git a/src/include/AutoModel/modeling_lfm2.hpp b/src/include/AutoModel/modeling_lfm2.hpp index 432d9971e..7567eccdd 100644 --- a/src/include/AutoModel/modeling_lfm2.hpp +++ b/src/include/AutoModel/modeling_lfm2.hpp @@ -26,7 +26,7 @@ class LFM2 : public AutoModel { public: LFM2(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; @@ -58,7 +58,7 @@ class LFM2_5_TK : public AutoModel { public: LFM2_5_TK(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; diff --git a/src/include/AutoModel/modeling_llama3.hpp b/src/include/AutoModel/modeling_llama3.hpp index ddf903a9b..510979de2 100644 --- a/src/include/AutoModel/modeling_llama3.hpp +++ b/src/include/AutoModel/modeling_llama3.hpp @@ -16,7 +16,7 @@ class Llama3 : public AutoModel { public: Llama3(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; @@ -41,7 +41,7 @@ class DeepSeek_r1_8b : public AutoModel { public: DeepSeek_r1_8b(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; diff --git a/src/include/AutoModel/modeling_nanbeige.hpp b/src/include/AutoModel/modeling_nanbeige.hpp index 69bad87c2..09b8c7ced 100644 --- a/src/include/AutoModel/modeling_nanbeige.hpp +++ b/src/include/AutoModel/modeling_nanbeige.hpp @@ -18,7 +18,7 @@ class Nanbeige : public AutoModel { public: Nanbeige(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; diff --git a/src/include/AutoModel/modeling_phi4.hpp b/src/include/AutoModel/modeling_phi4.hpp index 0622ee56d..56d15499b 100644 --- a/src/include/AutoModel/modeling_phi4.hpp +++ b/src/include/AutoModel/modeling_phi4.hpp @@ -1,27 +1,55 @@ -/// \file phi4.hpp -/// \brief phi4 class -/// \author FastFlowLM Team -/// \date 2025-09-04 -/// \version 0.9.25 -/// \note This is a source file for the phi4 class +/// \file modeling_phi4.hpp +/// \brief Phi-4 frontend +/// \note Phi-4 runs on either of two engines -- FastFlowLM's own NPU kernels or +/// ryzenai-corelib -- but this class knows nothing about either. +/// Which engine to build, and every rule for driving it, lives behind +/// flm::backend::ModelBackend; see AutoModel/model_backend.hpp. #pragma once #include "AutoModel/automodel.hpp" -/************ phi4 family **************/ +#if defined(FLM_CORELIB_TESTING) +namespace flm::phi4::testing { class Phi4FrontendTestAccess; } +#endif + class Phi4 : public AutoModel { private: - void setup_tokenizer(std::string model_path); +#if defined(FLM_CORELIB_TESTING) + /// \note The tokenizer contract is only observable from inside the class, + /// and the suite that checks it is not allowed to change it. + friend class flm::phi4::testing::Phi4FrontendTestAccess; +#endif + + /// \brief Build the tokenizer, chat template and stop ids + /// \param model_path the model directory + /// \note Phi-4's contract differs from the shared one: minja receives no + /// textual BOS/EOS, and there is no automatic BOS token. + /// \brief build the chat template and stop ids from an already-parsed + /// tokenizer_config.json + /// \param config the parse load_model handed to the backend as well + void setup_tokenizer(const nlohmann::json& config); + + /// \brief Turn a failed inference into a request error, clearing the session + /// \throws ModelRequestError 500, always + [[noreturn]] void fail_inference(); public: - Phi4(flm_rt::device* npu_device_inst); + explicit Phi4(flm_rt::device* npu_device_inst); + void load_model(std::string model_path, json model_info, + int default_context_length = -1, + bool enable_preemption = false, const std::string& backend = "") override; + bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, + std::function is_cancelled = [] { return false; }) override; + std::string generate(chat_meta_info_t& meta_info, int length_limit, + std::ostream& os, + std::function is_cancelled = [] { return false; }) override; + std::string generate_with_prompt(chat_meta_info_t& meta_info, + lm_uniform_input_t& input, + int length_limit, + std::ostream& os = std::cout) override; + std::string apply_chat_template(nlohmann::ordered_json& messages, + nlohmann::ordered_json tools = nlohmann::ordered_json::object()) override; - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; - //void toggle_enable_think() override; - bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; - std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; - std::string generate_with_prompt(chat_meta_info_t& meta_info, lm_uniform_input_t& input, int length_limit, std::ostream& os = std::cout) override; - std::string apply_chat_template(nlohmann::ordered_json& messages, nlohmann::ordered_json tools = nlohmann::ordered_json::object()) override; - bool check_using_checkpint() { - return false; - } + /// \note Phi-4 does not reuse a checkpointed prefix, so a matched prefix + /// still costs one round. Overrides the base default of true. + bool check_using_checkpint() override { return false; } }; diff --git a/src/include/AutoModel/modeling_qwen2.hpp b/src/include/AutoModel/modeling_qwen2.hpp index bc37d19bc..b0439bc43 100644 --- a/src/include/AutoModel/modeling_qwen2.hpp +++ b/src/include/AutoModel/modeling_qwen2.hpp @@ -19,7 +19,7 @@ class Qwen2 : public AutoModel { public: Qwen2(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; diff --git a/src/include/AutoModel/modeling_qwen2vl.hpp b/src/include/AutoModel/modeling_qwen2vl.hpp index bb4d8e78d..1a2e05347 100644 --- a/src/include/AutoModel/modeling_qwen2vl.hpp +++ b/src/include/AutoModel/modeling_qwen2vl.hpp @@ -45,7 +45,7 @@ class Qwen2VL : public AutoModel { public: Qwen2VL(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; diff --git a/src/include/AutoModel/modeling_qwen3.hpp b/src/include/AutoModel/modeling_qwen3.hpp index 3dd6dbec9..ba96d8d1e 100644 --- a/src/include/AutoModel/modeling_qwen3.hpp +++ b/src/include/AutoModel/modeling_qwen3.hpp @@ -26,7 +26,7 @@ class Qwen3 : public AutoModel { public: Qwen3(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; @@ -88,7 +88,7 @@ class Qwen3_IT : public AutoModel { public: Qwen3_IT(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; @@ -121,7 +121,7 @@ class Qwen3_TK : public AutoModel { public: Qwen3_TK(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; @@ -146,7 +146,7 @@ class DeepSeek_r1_0528_8b : public AutoModel { public: DeepSeek_r1_0528_8b(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; diff --git a/src/include/AutoModel/modeling_qwen3_5_omni.hpp b/src/include/AutoModel/modeling_qwen3_5_omni.hpp index b25491259..54e744090 100644 --- a/src/include/AutoModel/modeling_qwen3_5_omni.hpp +++ b/src/include/AutoModel/modeling_qwen3_5_omni.hpp @@ -32,7 +32,7 @@ class Qwen3_5_Omni : public AutoModel { ~Qwen3_5_Omni() override = default; /// \brief Load config + weights and set up tokenizer / sampler. - void load_model(std::string model_path, json model_info, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_info, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; /// \brief Apply the chat template to the messages. std::string apply_chat_template(nlohmann::ordered_json& messages, nlohmann::ordered_json tools = nlohmann::ordered_json::object()) override; diff --git a/src/include/AutoModel/modeling_qwen3_5vl.hpp b/src/include/AutoModel/modeling_qwen3_5vl.hpp index 9a9277758..430ad1bfd 100644 --- a/src/include/AutoModel/modeling_qwen3_5vl.hpp +++ b/src/include/AutoModel/modeling_qwen3_5vl.hpp @@ -52,7 +52,7 @@ class Qwen3_5VL : public AutoModel { public: Qwen3_5VL(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; diff --git a/src/include/AutoModel/modeling_qwen3_6_moe.hpp b/src/include/AutoModel/modeling_qwen3_6_moe.hpp index bc3b26ce9..0887e73f0 100644 --- a/src/include/AutoModel/modeling_qwen3_6_moe.hpp +++ b/src/include/AutoModel/modeling_qwen3_6_moe.hpp @@ -52,7 +52,7 @@ class Qwen3_6_MOE : public AutoModel { public: Qwen3_6_MOE(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; diff --git a/src/include/AutoModel/modeling_qwen3vl.hpp b/src/include/AutoModel/modeling_qwen3vl.hpp index 6e6311a3f..7a37a2231 100644 --- a/src/include/AutoModel/modeling_qwen3vl.hpp +++ b/src/include/AutoModel/modeling_qwen3vl.hpp @@ -61,12 +61,13 @@ class Qwen3VL : public AutoModel { qwen3vl_image_payload_t& image_payload, std::vector& tokens); - /// \brief Build the engine that backs this wrapper. - /// \note The Qwen3-VL checkpoint is served by two engines that share this - /// whole wrapper -- the tokenizer, chat template, sampler and image - /// preprocessing are identical -- and differ only in how they run - /// prefill on the NPU. This is the single seam between them. - virtual void create_engine(); + /// \note The Qwen3-VL checkpoint is served by two engines that share this + /// whole wrapper -- the tokenizer, chat template, sampler and image + /// preprocessing are identical -- and differ only in how they run + /// prefill on the NPU. Which one to build is no longer a seam here: + /// the two are separate families in the catalog ("qwen3vl" and + /// "qwen3vl-flash") and each registers its own backend, so the + /// registry picks the engine. See AutoModel/builtin_backends.cpp. /// \brief Longer-side target (pixels) that load_image/load_image_base64 must /// resize every decoded image to, applied before the user-configured @@ -76,7 +77,7 @@ class Qwen3VL : public AutoModel { public: Qwen3VL(flm_rt::device* npu_device_inst); - void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false) override; + void load_model(std::string model_path, json model_inf, int default_context_length = -1, bool enable_preemption = false, const std::string& backend = "") override; //void toggle_enable_think() override; bool insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, std::function is_cancelled = [] { return false; }) override; std::string generate(chat_meta_info_t& meta_info, int length_limit, std::ostream& os, std::function is_cancelled = [] { return false; }) override; @@ -176,8 +177,6 @@ class Qwen3VL_Flash : public Qwen3VL { int _pin_system_prefix(const std::string& system_text); protected: - void create_engine() override; - /// \brief The flash engine is tuned for short contexts, so every image is /// forced down to a 256px longer side regardless of img_pre_resize. int _forced_long_side() const override { return 256; } diff --git a/src/include/lm_config.hpp b/src/include/lm_config.hpp index ea1cb30aa..e98358930 100644 --- a/src/include/lm_config.hpp +++ b/src/include/lm_config.hpp @@ -8,6 +8,7 @@ #include "typedef.hpp" #include "utils/utils.hpp" +#include "utils/file_access.hpp" #include "nlohmann/json.hpp" #include @@ -99,7 +100,9 @@ class LM_Config{ /// \brief read model_path/config.json into _json_config void _load_json(){ - std::ifstream file(this->model_path + "/config.json"); + const auto config_path = std::filesystem::path(this->model_path) / "config.json"; + flm::file_access::ObserveOpen(config_path); + std::ifstream file(config_path); if (!file.is_open()){ std::cerr << "Failed to open file: " << this->model_path << std::endl; exit(1); diff --git a/src/include/model_list.hpp b/src/include/model_list.hpp index 7ea11e554..9e115c049 100644 --- a/src/include/model_list.hpp +++ b/src/include/model_list.hpp @@ -1,233 +1,354 @@ -/// \file model_list.hpp -/// \brief model_list class -/// \author FastFlowLM Team -/// \date 2025-06-24 -/// \version 0.9.24 -/// \note This class is used to manage the model list. -#pragma once -#include "nlohmann/json.hpp" -#include -#include -#include -#include -#include -#include -#include -#include "utils/utils.hpp" - -/// \note This class is used to manage the model list. -class model_list { - public: - std::unordered_set all_tags; - /// \brief constructor - model_list(){} - - - /// \brief constructor - /// \param list_path the path to the model list - /// \param exe_dir the executable directory for resolving relative paths - model_list(std::string& list_path, std::string& exe_dir){ - this->list_path = list_path; - std::ifstream config_file(list_path); - if (!config_file.is_open()) { - std::cerr << "Failed to open config file: " << list_path << std::endl; - exit(1); - } - this->config = nlohmann::json::parse(config_file); - // Resolve model_root_path relative to executable directory - std::string relative_model_path = this->config["model_path"]; - std::filesystem::path root_path = std::filesystem::path(exe_dir) / relative_model_path; - this->model_root_path = root_path.string(); - config_file.close(); - - // Populate all_tags set - for (const auto& [model_type, sizes] : this->config["models"].items()) { - // insert default tag without size - all_tags.insert(model_type); - for (const auto& [size, model_info] : sizes.items()) { - all_tags.insert(model_type + ":" + size); - } - } - } - - /// \brief get the model info - /// \param tag the tag of the model - /// \return the model info - std::pair get_model_info(const std::string tag) const { - static std::string last_error_tag = ""; - std::string new_tag = rectify_model_tag(tag); - bool model_found = false; - // get model type, the string before ':' in the tag - std::string model_type; - std::string model_size; - - if (new_tag.find(':') != std::string::npos) { - model_type = new_tag.substr(0, new_tag.find(':')); - model_size = new_tag.substr(new_tag.find(':') + 1); - } - else { - model_type = new_tag; - model_size = ""; - } - - // find the model subset first, compare with the key of the model - bool model_subset_found = false; - for (const auto& [key, model] : this->config["models"].items()) { - if (key == model_type) { - model_subset_found = true; - break; - } - } - if (model_subset_found) { - bool model_found = false; - for (const auto& [key, model] : this->config["models"][model_type].items()) { - if (key == model_size) { // if the size is found, return the model - model_found = true; - return std::make_pair(new_tag, model); - } - } - if (!model_found) { - if (last_error_tag != new_tag) { - last_error_tag = new_tag; - header_print_r("ERROR", "Model not found: " + model_size + " in subset " + model_type); - header_print_r("ERROR", "Using default model: llama3.2-1B"); - } - return std::make_pair("llama3.2:1b", this->config["models"]["llama3.2"]["1b"]); - } - } - else{ - if (last_error_tag != new_tag) { - last_error_tag = new_tag; - header_print_r("ERROR", "Model subset not found: " << model_type << "; using default model: llama3.2-1B"); - } - return std::make_pair("llama3.2:1b", this->config["models"]["llama3.2"]["1b"]); - } - return std::make_pair("llama3.2:1b", this->config["models"]["llama3.2"]["1b"]); - } - - /// \brief cut the tag, some program adds a prefix to the tag, like "Ollama/llama3.2-1B", we need to cut the prefix - /// \param tag the tag of the model - /// \return the model type, string - std::string cut_tag(const std::string tag) const { - std::string new_tag = tag; - if (tag.find('/') != std::string::npos) { - new_tag = tag.substr(tag.find('/') + 1); - } - return new_tag; - } - - /// \brief rectify the model tag, remove / and replace with actuall tag if size is not specified - /// \param original_tag the original tag of the model - /// \return the rectified model tag, string - std::string rectify_model_tag(const std::string original_tag) const { - std::string new_tag = this->cut_tag(original_tag); - // check if size is specified - if (new_tag.find(':') == std::string::npos) { - // get the first size in the subset - std::string model_type = new_tag; - std::string model_size = this->config["models"][model_type].begin().key(); - new_tag = model_type + ":" + model_size; - } - return new_tag; - } - - /// \brief get the model root path - /// \return the model root path, string - std::string get_model_root_path(){ - return this->model_root_path; - } - - /// \brief get all the models - /// \return all the models in json - nlohmann::json get_all_models(){ - nlohmann::json response = { - {"models", nlohmann::json::array()} - }; - - for (const auto& [model_type, model_subset] : this->config["models"].items()) { - for (const auto& [size, model_info] : model_subset.items()) { - nlohmann::json model_entry = model_info; - model_entry["name"] = model_type + ":" + size; - model_entry["model"] = model_type + ":" + size; - response["models"].push_back(model_entry); - } - } - return response; - } - - /// \brief get all the models - /// \return all the models in json - nlohmann::json get_all_models_ollama() { - nlohmann::json response = { - {"models", nlohmann::json::array()} - }; - - for (const auto& [model_type, model_subset] : this->config["models"].items()) { - if (model_type == "whisper-v3") continue; - else if (model_type == "embed-gemma") continue; - for (const auto& [size, model_info] : model_subset.items()) { - nlohmann::json model_entry = { - {"name", model_type + ":" + size}, - {"model", model_type + ":" + size}, - {"details", { - {"family", model_info["details"]["family"]}, - {"parameter_size", model_info["details"]["parameter_size"]}, - {"quantization_level", model_info["details"]["quantization_level"]} - }} - }; - response["models"].push_back(model_entry); - } - } - return response; - } - - /// \brief get all the models - /// \return all the models in json - nlohmann::json get_all_models_openai() { - nlohmann::json response = { - {"object", "list"}, - {"data", nlohmann::json::array()}, - {"object", "list" } - }; - - std::time_t now = std::time(nullptr); - - for (const auto& [model_type, model_subset] : this->config["models"].items()) { - if (model_type == "whisper-v3") continue; - else if (model_type == "embed-gemma") continue; - for (const auto& [size, model_info] : model_subset.items()) { - // id uses the same "type:size" convention; created uses current epoch seconds - nlohmann::json model_entry = { - {"id", model_type + ":" + size}, - {"object", "model"}, - {"created", static_cast(now)}, - {"owned_by", "FastFlowLM"} - }; - response["data"].push_back(model_entry); - } - } - - return response; - } - - /// \brief get the model path - /// \param tag the tag of the model - /// \return the model path, string - std::string get_model_path(const std::string& tag){ - std::string new_tag = this->rectify_model_tag(tag); - auto [new_tag_unused, model_info] = this->get_model_info(new_tag); - std::string model_name = model_info["name"]; - std::filesystem::path full_path = std::filesystem::path(this->model_root_path) / model_name; - return full_path.string(); - } - - bool is_model_supported(const std::string& tag) { - return all_tags.find(tag) != all_tags.end(); - } - - private: - std::string list_path; - nlohmann::json config; - std::string model_root_path; - -}; +/// \file model_list.hpp +/// \brief model_list class +/// \author FastFlowLM Team +/// \date 2025-06-24 +/// \version 0.9.24 +/// \note This class is used to manage the model list. +#pragma once +#include "nlohmann/json.hpp" +#include +#include +#include +#include +#include +#include +#include +#include +// Only the header_print* macros are needed here. Including utils/utils.hpp +// instead would drag in the NPU runtime headers and make this class impossible +// to unit test without an XRT/HRX toolchain. +#include "utils/debug_utils.hpp" + +/// \note This class is used to manage the model list. +class model_list { + public: + std::unordered_set all_tags; + /// \brief constructor + model_list(){} + + + /// \brief constructor + /// \param list_path the path to the model list + /// \param exe_dir the executable directory for resolving relative paths + /// \param platform the detected NPU generation ("stx" or "aie_next"); the + /// catalog is pruned to the models that generation can run, and + /// each surviving entry has its platform_overrides patch applied + model_list(std::string& list_path, std::string& exe_dir, + std::string platform = "stx"){ + this->list_path = list_path; + this->platform_ = std::move(platform); + std::ifstream config_file(list_path); + if (!config_file.is_open()) { + std::cerr << "Failed to open config file: " << list_path << std::endl; + exit(1); + } + this->config = nlohmann::json::parse(config_file); + // Resolve model_root_path relative to executable directory + std::string relative_model_path = this->config["model_path"]; + std::filesystem::path root_path = std::filesystem::path(exe_dir) / relative_model_path; + this->model_root_path = root_path.string(); + config_file.close(); + + // Prune before indexing: all_tags must describe what this machine can + // actually run, so an unsupported tag fails at validation instead of + // failing much later inside the model backend. + this->apply_platform_filter(); + + // Populate all_tags set + for (const auto& [model_type, sizes] : this->config["models"].items()) { + // insert default tag without size + all_tags.insert(model_type); + for (const auto& [size, model_info] : sizes.items()) { + all_tags.insert(model_type + ":" + size); + } + } + + if (all_tags.empty()) { + header_print_r("ERROR", "No models in " + this->list_path + + " support this NPU (" + this->platform_ + ")"); + exit(1); + } + } + + /// \brief the NPU generation this catalog was filtered for + /// \return "stx" or "aie_next" + const std::string& platform() const { return this->platform_; } + + /// \brief get the model info + /// \param tag the tag of the model + /// \return the model info + std::pair get_model_info(const std::string tag) const { + static std::string last_error_tag = ""; + std::string new_tag = rectify_model_tag(tag); + bool model_found = false; + // get model type, the string before ':' in the tag + std::string model_type; + std::string model_size; + + if (new_tag.find(':') != std::string::npos) { + model_type = new_tag.substr(0, new_tag.find(':')); + model_size = new_tag.substr(new_tag.find(':') + 1); + } + else { + model_type = new_tag; + model_size = ""; + } + + // find the model subset first, compare with the key of the model + bool model_subset_found = false; + for (const auto& [key, model] : this->config["models"].items()) { + if (key == model_type) { + model_subset_found = true; + break; + } + } + if (model_subset_found) { + bool model_found = false; + for (const auto& [key, model] : this->config["models"][model_type].items()) { + if (key == model_size) { // if the size is found, return the model + model_found = true; + return std::make_pair(new_tag, model); + } + } + if (!model_found) { + auto fallback = this->fallback_model(); + if (last_error_tag != new_tag) { + last_error_tag = new_tag; + header_print_r("ERROR", "Model not found: " + model_size + " in subset " + model_type); + header_print_r("ERROR", "Using default model: " + fallback.first); + } + return fallback; + } + } + else{ + auto fallback = this->fallback_model(); + if (last_error_tag != new_tag) { + last_error_tag = new_tag; + header_print_r("ERROR", "Model subset not found: " << model_type << "; using default model: " << fallback.first); + } + return fallback; + } + return this->fallback_model(); + } + + /// \brief cut the tag, some program adds a prefix to the tag, like "Ollama/llama3.2-1B", we need to cut the prefix + /// \param tag the tag of the model + /// \return the model type, string + std::string cut_tag(const std::string tag) const { + std::string new_tag = tag; + if (tag.find('/') != std::string::npos) { + new_tag = tag.substr(tag.find('/') + 1); + } + return new_tag; + } + + /// \brief rectify the model tag, remove / and replace with actuall tag if size is not specified + /// \param original_tag the original tag of the model + /// \return the rectified model tag, string + std::string rectify_model_tag(const std::string original_tag) const { + std::string new_tag = this->cut_tag(original_tag); + // check if size is specified + if (new_tag.find(':') == std::string::npos) { + const std::string model_type = new_tag; + // A family pruned for this platform (or simply misspelled) has no + // sizes to pick from. Return the tag untouched so get_model_info + // reports it rather than dereferencing a null subset. + const auto& models = this->config["models"]; + if (!models.contains(model_type) || models.at(model_type).empty()) { + return new_tag; + } + // get the first size in the subset + std::string model_size = models.at(model_type).begin().key(); + new_tag = model_type + ":" + model_size; + } + return new_tag; + } + + /// \brief get the model root path + /// \return the model root path, string + std::string get_model_root_path(){ + return this->model_root_path; + } + + /// \brief get all the models + /// \return all the models in json + nlohmann::json get_all_models(){ + nlohmann::json response = { + {"models", nlohmann::json::array()} + }; + + for (const auto& [model_type, model_subset] : this->config["models"].items()) { + for (const auto& [size, model_info] : model_subset.items()) { + nlohmann::json model_entry = model_info; + model_entry["name"] = model_type + ":" + size; + model_entry["model"] = model_type + ":" + size; + response["models"].push_back(model_entry); + } + } + return response; + } + + /// \brief get all the models + /// \return all the models in json + nlohmann::json get_all_models_ollama() { + nlohmann::json response = { + {"models", nlohmann::json::array()} + }; + + for (const auto& [model_type, model_subset] : this->config["models"].items()) { + if (model_type == "whisper-v3") continue; + else if (model_type == "embed-gemma") continue; + for (const auto& [size, model_info] : model_subset.items()) { + nlohmann::json model_entry = { + {"name", model_type + ":" + size}, + {"model", model_type + ":" + size}, + {"details", { + {"family", model_info["details"]["family"]}, + {"parameter_size", model_info["details"]["parameter_size"]}, + {"quantization_level", model_info["details"]["quantization_level"]} + }} + }; + response["models"].push_back(model_entry); + } + } + return response; + } + + /// \brief get all the models + /// \return all the models in json + nlohmann::json get_all_models_openai() { + nlohmann::json response = { + {"object", "list"}, + {"data", nlohmann::json::array()}, + {"object", "list" } + }; + + std::time_t now = std::time(nullptr); + + for (const auto& [model_type, model_subset] : this->config["models"].items()) { + if (model_type == "whisper-v3") continue; + else if (model_type == "embed-gemma") continue; + for (const auto& [size, model_info] : model_subset.items()) { + // id uses the same "type:size" convention; created uses current epoch seconds + nlohmann::json model_entry = { + {"id", model_type + ":" + size}, + {"object", "model"}, + {"created", static_cast(now)}, + {"owned_by", "FastFlowLM"} + }; + response["data"].push_back(model_entry); + } + } + + return response; + } + + /// \brief get the model path + /// \param tag the tag of the model + /// \return the model path, string + std::string get_model_path(const std::string& tag){ + std::string new_tag = this->rectify_model_tag(tag); + auto [new_tag_unused, model_info] = this->get_model_info(new_tag); + std::string model_name = model_info["name"]; + std::filesystem::path full_path = std::filesystem::path(this->model_root_path) / model_name; + return full_path.string(); + } + + bool is_model_supported(const std::string& tag) { + return all_tags.find(tag) != all_tags.end(); + } + + private: + std::string list_path; + nlohmann::json config; + std::string model_root_path; + std::string platform_; + + /// \brief whether an entry claims support for the active platform + /// \param entry the size entry + /// \param tag the "family:size" tag, used only in error messages + /// \return true if the entry runs on this->platform_ + /// \note An entry that says nothing is stx-only. stx is what every + /// model runs on, so the catalog only tags the exceptions: an + /// entry needs "supported_platforms" exactly when it runs on aie_next. + bool entry_supports(const nlohmann::json& entry, const std::string& tag) const { + const auto supported = entry.find("supported_platforms"); + if (supported == entry.end()) return this->platform_ == "stx"; + if (!supported->is_array() || supported->empty()) { + throw std::runtime_error( + "supported_platforms must be a non-empty array: " + tag); + } + for (const auto& value : *supported) { + if (!value.is_string()) { + throw std::runtime_error( + "supported_platforms must contain strings: " + tag); + } + if (value.get() == this->platform_) return true; + } + return false; + } + + /// \brief drop entries this NPU cannot run and flatten the survivors + /// \note After this runs the config has exactly the shape it had before + /// platform support existed, so nothing downstream needs to know + /// which platform was selected. + void apply_platform_filter() { + std::vector empty_families; + + for (auto& [model_type, model_subset] : this->config["models"].items()) { + std::vector unsupported_sizes; + + for (auto& [size, model_info] : model_subset.items()) { + const std::string tag = model_type + ":" + size; + if (!entry_supports(model_info, tag)) { + unsupported_sizes.push_back(size); + continue; + } + // Take the patch first, then erase the bookkeeping keys, so a + // malformed override can never reintroduce them. + nlohmann::json patch = nlohmann::json::object(); + const auto overrides = model_info.find("platform_overrides"); + if (overrides != model_info.end()) { + if (!overrides->is_object()) { + throw std::runtime_error( + "platform_overrides must be an object: " + tag); + } + const auto match = overrides->find(this->platform_); + if (match != overrides->end()) patch = *match; + } + model_info.erase("platform_overrides"); + model_info.erase("supported_platforms"); + // merge_patch replaces arrays wholesale, which is what "files" + // needs, and a null value deletes the key (e.g. "ms_url"). + if (!patch.empty()) model_info.merge_patch(patch); + } + + for (const auto& size : unsupported_sizes) model_subset.erase(size); + if (model_subset.empty()) empty_families.push_back(model_type); + } + + for (const auto& model_type : empty_families) { + this->config["models"].erase(model_type); + } + } + + /// \brief the entry to fall back on when a tag cannot be resolved + /// \return the fallback tag and its info + /// \note llama3.2:1b is the historical default, but it is pruned on + /// platforms that cannot run it, so fall back to whatever survived. + std::pair fallback_model() const { + const auto& models = this->config["models"]; + if (models.contains("llama3.2") && + models["llama3.2"].contains("1b")) { + return std::make_pair("llama3.2:1b", models["llama3.2"]["1b"]); + } + for (const auto& [model_type, model_subset] : models.items()) { + for (const auto& [size, model_info] : model_subset.items()) { + return std::make_pair(model_type + ":" + size, model_info); + } + } + throw std::runtime_error("No models available for NPU platform " + + this->platform_); + } + +}; diff --git a/src/include/models/phi4/rai/phi4_rai.hpp b/src/include/models/phi4/rai/phi4_rai.hpp new file mode 100644 index 000000000..50903465b --- /dev/null +++ b/src/include/models/phi4/rai/phi4_rai.hpp @@ -0,0 +1,44 @@ +#pragma once + +#include "causal_lm.hpp" +#include "rai/corelib_runtime.hpp" +#include "lm_config.hpp" +#include "models/phi4/rai/phi4_rai_gguf.hpp" + +#include +#include + +namespace flm::phi4 { + +class phi4_rai final : public causal_lm { +public: + phi4_rai( + LM_Config config, + std::shared_ptr package, + std::shared_ptr runtime, + std::uint32_t max_length = 4096); + ~phi4_rai() override; + + buffer forward(int id) override; + buffer prefill(std::vector& ids, void* payload = nullptr) override; + void set_context_length(int length) override; + /// \brief unsupported; this engine's weights come from its GGUF package + /// \note Present only because causal_lm.hpp is a frozen ABI. See the + /// definition and AutoModel/model_backend.hpp. + /// \throws std::runtime_error always + void load_weights(Q4NX&) override; + void update_max_length(std::uint32_t max_length) override; + void clear_context() override; + buffer get_k_cache(int layer, int index) override; + buffer get_v_cache(int layer, int index) override; + int get_current_context_length() override; + int checkpoint() override; + int restore() override; + bool poisoned() const noexcept; + +private: + struct Impl; + std::unique_ptr impl_; +}; + +} // namespace flm::phi4 diff --git a/src/include/models/phi4/rai/phi4_rai_backend.hpp b/src/include/models/phi4/rai/phi4_rai_backend.hpp new file mode 100644 index 000000000..6609ec4c8 --- /dev/null +++ b/src/include/models/phi4/rai/phi4_rai_backend.hpp @@ -0,0 +1,38 @@ +/// \file phi4_rai_backend.hpp +/// \brief The ryzenai-corelib backend for Phi-4 +/// \note Compiled only when FLM_ENABLE_RAI is on. Everything that used +/// to be a rai special case inside the Phi4 frontend -- the 4095-token +/// decode limit, the no-preemption rule, the package-verified EOS ids, the +/// poisoning state -- is stated here instead. +#pragma once + +#include "AutoModel/model_backend.hpp" + +#include + +namespace flm::phi4 { + +/// \brief the largest context this backend can hold +inline constexpr std::uint32_t kRaiContextLimit = 4096; + +/// \brief the largest number of tokens corelib will decode +inline constexpr std::uint32_t kRaiDecodeLimit = 4095; + +/// \brief what the frontend must know before building this backend +/// \return no xclbin, no preemption, 4096-token context ceiling +/// \note Pure data, and inline on purpose: the registry needs it to reject a +/// request before the engine exists, and so does anything standing in for +/// the engine. +inline flm::backend::BackendTraits rai_traits() { + flm::backend::BackendTraits traits; + traits.needs_npu_xclbin = false; + traits.supports_preemption = false; + traits.max_context_length = kRaiContextLimit; + return traits; +} + +/// \brief a factory building the rai backend +/// \return a factory suitable for BackendRegistry::register_backend +flm::backend::BackendFactory rai_factory(); + +} // namespace flm::phi4 diff --git a/src/include/models/phi4/rai/phi4_rai_constants.hpp b/src/include/models/phi4/rai/phi4_rai_constants.hpp new file mode 100644 index 000000000..7213647ed --- /dev/null +++ b/src/include/models/phi4/rai/phi4_rai_constants.hpp @@ -0,0 +1,44 @@ +#pragma once + +#include + +namespace flm::phi4 { +inline constexpr std::int64_t kLayerCount = 32; +inline constexpr std::int64_t kHiddenSize = 3072; +inline constexpr std::int64_t kIntermediateSize = 8192; +inline constexpr std::int64_t kQueryHeadCount = 24; +inline constexpr std::int64_t kKvHeadCount = 8; +inline constexpr std::int64_t kHeadSize = 128; +inline constexpr std::int64_t kQueryDimension = 3072; +inline constexpr std::int64_t kKvDimension = 1024; +inline constexpr std::int64_t kVocabularySize = 200064; +inline constexpr std::int64_t kRopeDimension = 96; +inline constexpr std::int64_t kMaxSequenceLength = 4096; +inline constexpr std::int64_t kModelContextLength = 131072; +inline constexpr std::int64_t kMaxDecodeWindow = 4095; +inline constexpr std::uint32_t kRequantizedGroupSize = 64; +/// Intra-packer thread hint for one Q8_0 requantizing create. corelib treats 0 +/// as ONE deliberately. Packing many weights at once is the bigger lever and +/// belongs to the caller, so the parallelism is taken below as concurrent +/// creates instead; asking for both would oversubscribe the machine. +inline constexpr std::uint32_t kRequantizeThreads = 0; + +/// How many weight creates run at once. The 161 creates are independent -- each +/// reads its own mapped range of the GGUF and produces its own object -- so +/// this is the parallelism that actually shortens load. Real threads rather +/// than a packer hint, so it is not subject to whatever thread limits the +/// surrounding environment imposes on the packer. +inline constexpr std::size_t kWeightCreateConcurrency = 8; +inline constexpr float kRmsEpsilon = 1.0e-5f; + +/// The PDI pair this model's artifacts were built for, required by +/// ryzenai_corelib_create_stream since 0.5.0 and deliberately given no default +/// by corelib: the silicon ships the same operator under several PDI tags and a model's +/// ELFs exist under exactly one of them. Qwen3.6 is p9/p19; Phi-4 is one of the +/// older families, which are p1/p16. A stream opened on the wrong pair silently +/// loses every shape its pair is the only home of, and the loss surfaces as a +/// missing artifact rather than as a wrong answer -- so this is pinned here with +/// the rest of the model's facts rather than defaulted anywhere. +inline constexpr int kPrefillPdi = 1; +inline constexpr int kTokenPdi = 16; +} // namespace flm::phi4 diff --git a/src/include/models/phi4/rai/phi4_rai_gguf.hpp b/src/include/models/phi4/rai/phi4_rai_gguf.hpp new file mode 100644 index 000000000..436770c5f --- /dev/null +++ b/src/include/models/phi4/rai/phi4_rai_gguf.hpp @@ -0,0 +1,80 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +namespace flm::phi4 { + +struct TensorView { + std::string_view name; + std::span bytes; + std::vector logical_shape; + std::uint32_t ggml_type; +}; + +struct FloatTensorView { + std::string_view name; + std::span values; + std::vector logical_shape; +}; + +struct ProjectionViews { + std::array values; + std::size_t count; +}; + +struct GgufPhi4Metadata { + std::string architecture; + std::uint64_t layer_count; + std::uint64_t hidden_size; + std::uint64_t intermediate_size; + std::uint64_t attention_head_count; + std::uint64_t kv_head_count; + std::uint64_t context_length; + std::uint64_t rope_dimension_count; + double rope_frequency_base; + double rope_attention_factor; + std::uint64_t rope_original_context_length; + std::uint64_t tokenizer_vocabulary_size; + bool add_bos_token; +}; + +class Phi4GgufPackage final { +public: + static std::shared_ptr Open( + const std::filesystem::path& gguf_path); + ~Phi4GgufPackage(); + + TensorView RequireQ8( + std::string_view name, + std::span expected_shape) const; + FloatTensorView RequireF32( + std::string_view name, + std::span expected_shape) const; + ProjectionViews AttentionQkv(std::size_t layer) const; + ProjectionViews GateUp(std::size_t layer) const; + GgufPhi4Metadata Metadata() const; + /// \brief the file this package was opened from + /// \note Used to key the packed-weight cache on the GGUF's identity. + const std::filesystem::path& Path() const; + void ValidatePhi4Contract( + const nlohmann::json& config, + const nlohmann::json& tokenizer, + const nlohmann::json& tokenizer_config) const; + +private: + struct Impl; + explicit Phi4GgufPackage(std::unique_ptr impl); + std::unique_ptr impl_; +}; + +} // namespace flm::phi4 diff --git a/src/include/models/phi4/rai/phi4_rai_host.hpp b/src/include/models/phi4/rai/phi4_rai_host.hpp new file mode 100644 index 000000000..dbe7bf768 --- /dev/null +++ b/src/include/models/phi4/rai/phi4_rai_host.hpp @@ -0,0 +1,35 @@ +#pragma once + +#include "models/phi4/rai/phi4_rai_gguf.hpp" + +#include +#include +#include +#include + +namespace flm::phi4 { + +struct RopeTables { + std::vector cosine; + std::vector sine; +}; + +std::vector DecodeEmbeddingRowsQ8( + const TensorView& embedding, + std::span token_ids); + +void HostRmsNorm( + std::span input, + std::span scale, + std::int64_t rows, + std::int64_t width, + float epsilon, + std::span output); + +std::vector ConvertF32ToBf16(std::span values); + +RopeTables BuildShortRopeTables( + const GgufPhi4Metadata& metadata, + std::optional short_factors); + +} // namespace flm::phi4 diff --git a/src/include/models/phi4/rai/phi4_rai_shape_plan.hpp b/src/include/models/phi4/rai/phi4_rai_shape_plan.hpp new file mode 100644 index 000000000..afd246aa9 --- /dev/null +++ b/src/include/models/phi4/rai/phi4_rai_shape_plan.hpp @@ -0,0 +1,45 @@ +#pragma once + +#include "rai/corelib_api.hpp" + +#include +#include +#include +#include + +namespace flm::phi4 { + +struct Phi4RowExtents { + std::int64_t query_rows; + std::int64_t kv_rows; + std::int64_t output_rows; + std::int64_t ssmlp_rows; + std::int64_t flat_mha_rows; +}; + +class Phi4ShapePlan final { +public: + /// \brief interrogate corelib for the padded extent of every live row count + /// \param api the corelib binding + /// \param stream the stream the plan will be dispatched on + /// \note corelib 0.5.0 takes the stream on every padding helper: the PDI + /// pair a stream was opened with selects the ELF, and a shape exists + /// under one pair and not another. So the plan is only meaningful + /// for the stream it was built against, and cannot be built before + /// one exists. + static Phi4ShapePlan Build( + const std::shared_ptr& api, + ryzenai_corelib_stream_ptr stream); + const Phi4RowExtents& ForRows(std::size_t live_rows) const; + const Phi4RowExtents& maximum_extents() const noexcept; + const ryzenai_corelib_flat_mha_bf16_desc& attention_desc() const noexcept; + const ryzenai_corelib_matmul_bf16_weights_desc& lm_head_desc() const noexcept; + +private: + std::vector rows_; + Phi4RowExtents maximum_extents_{}; + ryzenai_corelib_flat_mha_bf16_desc attention_desc_{}; + ryzenai_corelib_matmul_bf16_weights_desc lm_head_desc_{}; +}; + +} // namespace flm::phi4 diff --git a/src/include/models/phi4/rai/phi4_rai_weight_cache.hpp b/src/include/models/phi4/rai/phi4_rai_weight_cache.hpp new file mode 100644 index 000000000..e460ef447 --- /dev/null +++ b/src/include/models/phi4/rai/phi4_rai_weight_cache.hpp @@ -0,0 +1,96 @@ +#pragma once + +/// \file phi4_rai_weight_cache.hpp +/// \brief On-disk cache of the packed weights, so requantization is paid once +/// \note Requantizing the 161 weights from Q8_0 is effectively the whole of +/// model load. corelib can hand the packed bytes back +/// (ryzenai_corelib_weights_copy_data) and load them again later +/// (..._weights_create_from_file, which maps rather than copies), so the +/// refit need only happen the first time a given GGUF is loaded. +/// +/// corelib's own guidance is that caching only pays for a large blob and +/// that for an ordinary matmul packing can be faster than reading a +/// precomputed one back. Whether it pays here is a measurement, which is +/// why the cache reports what it cost. + +#include +#include +#include +#include +#include + +namespace flm::phi4 { + +/// \brief where one packed weight lives inside the cache file +struct CachedWeightSpan { + std::uint64_t offset{}; + std::uint64_t size{}; +}; + +/// \brief the identity a cache is only valid for +/// \note Everything that can change the packed bytes has to be in here. The +/// GGUF is identified by size and write time rather than by content: the +/// content hash of a 4 GB file costs more than the packing the cache +/// exists to avoid, which would defeat it. corelib independently rejects +/// a slice whose length is not exactly what the descriptor packs to, so a +/// stale-but-plausible cache cannot quietly become wrong weights. +struct WeightCacheKey { + std::uint64_t gguf_size{}; + std::int64_t gguf_write_time{}; + std::uint32_t corelib_major{}, corelib_minor{}, corelib_patch{}; + std::uint32_t group_size{}; + std::uint64_t weight_count{}; + + bool operator==(const WeightCacheKey& other) const; +}; + +/// \brief an index read back from disk, when one is present and current +struct WeightCacheIndex { + WeightCacheKey key; + std::vector spans; +}; + +/// \brief the cache directory for a model, or nullopt when caching is disabled +/// \param model_path the directory the GGUF lives in +/// \return the directory to hold the cache, honouring FLM_RAI_WEIGHT_CACHE +/// \note Unset means the cache sits beside the model. Set to a path redirects +/// it; set to "0" or "off" disables caching entirely. +std::optional WeightCacheDirectory( + const std::filesystem::path& model_path); + +/// \brief compute the identity of the cache a given GGUF would produce +WeightCacheKey MakeWeightCacheKey(const std::filesystem::path& gguf_path, + std::uint32_t corelib_major, + std::uint32_t corelib_minor, + std::uint32_t corelib_patch, + std::uint32_t group_size, + std::uint64_t weight_count); + +/// \brief read the index beside a cache file, if it matches the expected key +/// \return the index, or nullopt when absent, unreadable or stale +/// \note Never throws: a damaged cache is a cache miss, not a failed load. +std::optional ReadWeightCacheIndex( + const std::filesystem::path& directory, const WeightCacheKey& expected); + +/// \brief write the index describing a freshly written cache file +/// \return true when the index landed, false when it could not be written +/// \note Written last and renamed into place, so a cache file without a +/// matching index is never mistaken for a usable one. +bool WriteWeightCacheIndex(const std::filesystem::path& directory, + const WeightCacheKey& key, + const std::vector& spans); + +/// \brief the cache file itself, beside its index +std::filesystem::path WeightCacheDataPath(const std::filesystem::path& directory); + +/// \brief delete a cache that is not going to be used +/// \param directory the cache directory +/// \return how many bytes were reclaimed +/// \note Called before repacking, so a cache that no longer matches its GGUF +/// stops occupying two gigabytes from the moment it is known to be +/// useless rather than from whenever the next write happens to succeed. +/// Also clears the temporaries an interrupted write leaves behind. +/// Never throws: failing to delete is not a reason to fail a load. +std::uint64_t RemoveWeightCache(const std::filesystem::path& directory); + +} // namespace flm::phi4 diff --git a/src/include/program_args.hpp b/src/include/program_args.hpp index 59ddfbe29..35313a85c 100644 --- a/src/include/program_args.hpp +++ b/src/include/program_args.hpp @@ -19,6 +19,7 @@ struct program_args_t { bool embed = false; bool json_output = false; int ctx_length = -1; // let model decide + std::string backend = ""; // let the catalog decide; see AutoModel/model_backend.hpp int prefill_chunk_len = -1; // let model decide // handling input file diff --git a/src/include/rai/corelib_api.hpp b/src/include/rai/corelib_api.hpp new file mode 100644 index 000000000..f6355bf44 --- /dev/null +++ b/src/include/rai/corelib_api.hpp @@ -0,0 +1,116 @@ +#pragma once + +#include + +#if RYZENAI_CORELIB_VERSION_MAJOR != 0 || RYZENAI_CORELIB_VERSION_MINOR != 5 || \ + RYZENAI_CORELIB_VERSION_PATCH != 0 +#error "FastFlowLM requires ryzenai-corelib headers exactly 0.5.0" +#endif + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define FLM_CORELIB_FUNCTIONS(X) \ + X(get_version, ryzenai_corelib_get_version) \ + X(status_to_string, ryzenai_corelib_status_to_string) \ + X(get_last_error_message, ryzenai_corelib_get_last_error_message) \ + X(selftest_dependencies, ryzenai_corelib_selftest_dependencies) \ + X(get_device, ryzenai_corelib_get_device) \ + X(object_release, ryzenai_corelib_object_release) \ + X(create_stream, ryzenai_corelib_create_stream) \ + X(stream_synchronize, ryzenai_corelib_stream_synchronize) \ + X(create_device_tensor, ryzenai_corelib_create_device_tensor) \ + X(create_tensor_window, ryzenai_corelib_create_tensor_window) \ + X(tensor_write, ryzenai_corelib_tensor_write) \ + X(tensor_read, ryzenai_corelib_tensor_read) \ + X(tensor_get_byte_size, ryzenai_corelib_tensor_get_byte_size) \ + X(tensor_get_data_type, ryzenai_corelib_tensor_get_data_type) \ + X(matmul_pad_shape, ryzenai_corelib_matmul_bf16_pad_shape) \ + X(matmul_weights_create_gguf_requantized, \ + ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized) \ + X(matmul_weights_create_from_file, \ + ryzenai_corelib_matmul_bf16_weights_create_from_file) \ + X(matmul, ryzenai_corelib_matmul_bf16) \ + X(ssmlp_pad_rows, ryzenai_corelib_ssmlp_bf16_pad_rows) \ + X(ssmlp_weights_create_gguf_requantized, \ + ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized) \ + X(ssmlp_weights_create_from_file, \ + ryzenai_corelib_ssmlp_bf16_weights_create_from_file) \ + X(weights_copy_data, ryzenai_corelib_weights_copy_data) \ + X(ssmlp, ryzenai_corelib_ssmlp_bf16) \ + X(flat_mha_pad_rows, ryzenai_corelib_flat_mha_bf16_pad_rows) \ + X(flat_mha, ryzenai_corelib_flat_mha_bf16) \ + X(cleanup, ryzenai_corelib_cleanup) + +namespace flm::corelib { + +struct CorelibVersion { + std::uint32_t major; + std::uint32_t minor; + std::uint32_t patch; +}; + +class CorelibError final : public std::runtime_error { +public: + CorelibError(ryzenai_corelib_status status, + std::string call, + std::string detail, + std::string status_text); + ryzenai_corelib_status status() const noexcept; + const std::string& call() const noexcept; + const std::string& detail() const noexcept; + +private: + ryzenai_corelib_status status_; + std::string call_; + std::string detail_; +}; + +struct CorelibFunctions { +#define FLM_DECLARE_CORELIB_FUNCTION(member, symbol) decltype(&::symbol) member{}; + FLM_CORELIB_FUNCTIONS(FLM_DECLARE_CORELIB_FUNCTION) +#undef FLM_DECLARE_CORELIB_FUNCTION +}; + +class CorelibApi final { +public: + using Resolver = std::function; + static std::shared_ptr Load(const std::filesystem::path& dll); +#if defined(FLM_CORELIB_LINK_STATIC) + /// \brief bind to the corelib linked into this binary + /// \return the API bound to the linked symbols + /// \note The static build has no DLL to pick, so FLM_RAI_CORELIB_PATH has + /// nothing to select and is reported as ignored. + static std::shared_ptr LoadStatic(); +#endif + static std::shared_ptr ResolveForTest( + Resolver resolver, std::filesystem::path loaded_library_path = {}); + static std::filesystem::path ResolveLibraryPath( + const std::filesystem::path& executable_dir); + const CorelibFunctions& functions() const noexcept; + CorelibVersion runtime_version() const noexcept; + const std::filesystem::path& loaded_library_path() const noexcept; + void Check(ryzenai_corelib_status status, std::string_view call) const; + void RegisterObject() const noexcept; + void Release(void* object) const noexcept; + std::size_t live_object_count() const noexcept; + +private: + explicit CorelibApi(Resolver resolver, + std::filesystem::path loaded_library_path = {}); + + Resolver resolver_; + CorelibFunctions functions_{}; + CorelibVersion runtime_version_{}; + std::filesystem::path loaded_library_path_; + mutable std::atomic live_object_count_{0}; +}; + +} // namespace flm::corelib diff --git a/src/include/rai/corelib_device.hpp b/src/include/rai/corelib_device.hpp new file mode 100644 index 000000000..557f52298 --- /dev/null +++ b/src/include/rai/corelib_device.hpp @@ -0,0 +1,25 @@ +/// \file corelib_device.hpp +/// \brief Access to the xrt::device that ryzenai-corelib owns. +/// \note FastFlowLM consumes corelib through the C ABI in +/// (see corelib_api.hpp), which has no device accessor: the device lives +/// behind corelib's C++ entry point ryzenai::corelib::GetDevice(). The +/// declaration is reproduced here rather than pulled from a corelib C++ +/// header so that this tree keeps depending on exactly one corelib header; +/// it resolves at link time against the statically linked corelib +/// (RYZENAI_CORELIB_STATIC), so a signature drift is a link error, not a +/// silent mismatch. +#pragma once + +#if defined(FLM_USE_HRX) +#error "FLM_ENABLE_RAI requires the XRT backend (FLM_USE_HRX=OFF)" +#endif + +#include "device_runtime.hpp" + +namespace ryzenai::corelib { + +/// \brief the device corelib initialized; valid until ryzenai_corelib_cleanup() +/// \return corelib's device, shared with every FLM engine +const xrt::device& GetDevice(); + +} // namespace ryzenai::corelib diff --git a/src/include/rai/corelib_object.hpp b/src/include/rai/corelib_object.hpp new file mode 100644 index 000000000..a8d92a109 --- /dev/null +++ b/src/include/rai/corelib_object.hpp @@ -0,0 +1,62 @@ +#pragma once + +#include "rai/corelib_api.hpp" + +#include +#include + +namespace flm::corelib { + +struct StreamTag {}; +struct TensorTag {}; +struct TensorWindowTag {}; +struct MatMulWeightsTag {}; +struct SsMlpWeightsTag {}; + +template +class UniqueObject final { +public: + UniqueObject() noexcept = default; + + UniqueObject(std::shared_ptr api, void* object) noexcept + : api_(std::move(api)), object_(object) { + if (object_) api_->RegisterObject(); + } + + ~UniqueObject() { reset(); } + + UniqueObject(const UniqueObject&) = delete; + UniqueObject& operator=(const UniqueObject&) = delete; + + UniqueObject(UniqueObject&& other) noexcept + : api_(std::move(other.api_)), object_(std::exchange(other.object_, nullptr)) {} + + UniqueObject& operator=(UniqueObject&& other) noexcept { + if (this != &other) { + reset(); + api_ = std::move(other.api_); + object_ = std::exchange(other.object_, nullptr); + } + return *this; + } + + void reset() noexcept { + if (object_) api_->Release(std::exchange(object_, nullptr)); + api_.reset(); + } + + void* get() const noexcept { return object_; } + explicit operator bool() const noexcept { return object_ != nullptr; } + +private: + std::shared_ptr api_; + void* object_{}; +}; + +using UniqueStream = UniqueObject; +using UniqueTensor = UniqueObject; +using UniqueTensorWindow = UniqueObject; +using UniqueMatMulWeights = UniqueObject; +using UniqueSsMlpWeights = UniqueObject; + +} // namespace flm::corelib diff --git a/src/include/rai/corelib_runtime.hpp b/src/include/rai/corelib_runtime.hpp new file mode 100644 index 000000000..33e096f66 --- /dev/null +++ b/src/include/rai/corelib_runtime.hpp @@ -0,0 +1,38 @@ +#pragma once + +#include "rai/corelib_api.hpp" + +#include +#include +#include +#if defined(FLM_CORELIB_TESTING) +#include +#endif + +namespace flm::corelib { + +class CorelibRuntime final { +public: + ~CorelibRuntime(); + static std::shared_ptr GetOrCreate( + const std::filesystem::path& executable_dir); + static std::shared_ptr CreateForTest( + std::shared_ptr api); + static void ShutdownProcess(); +#if defined(FLM_CORELIB_TESTING) + static void SetDestructionObserverForTest(std::function observer); +#endif + std::unique_lock AcquireExecution(); + const std::shared_ptr& api() const noexcept; + const std::filesystem::path& loaded_library_path() const noexcept; + +private: + explicit CorelibRuntime(std::shared_ptr api); + static std::shared_ptr CreateReady( + std::shared_ptr api); + + std::shared_ptr api_; + std::mutex execution_mutex_; +}; + +} // namespace flm::corelib diff --git a/src/include/utils/debug_utils.hpp b/src/include/utils/debug_utils.hpp index 9189b826f..6106ae213 100644 --- a/src/include/utils/debug_utils.hpp +++ b/src/include/utils/debug_utils.hpp @@ -5,6 +5,7 @@ /// \version 0.9.24 /// \note This file contains the debug utilities for the FastFlowLM project. #pragma once +#include #include #include #include @@ -181,3 +182,16 @@ inline std::string size_t_to_string(size_t size){ return std::to_string(size / (1024 * 1024 * 1024)) + "G"; } } + +/// \brief Report how long loading a model took. +/// \param started the time point captured immediately before load_model +/// \note Model load is the one phase no profiler covers, and on backends that +/// repack weights at load it dominates the time to a first usable prompt. Call +/// this from every path that loads a model so the CLI and the server agree. +inline void report_load_time(std::chrono::steady_clock::time_point started) { + const double seconds = + std::chrono::duration(std::chrono::steady_clock::now() - started).count(); + std::ostringstream message; + message << std::fixed << std::setprecision(2) << "Model loaded in " << seconds << " s"; + header_print("FLM", message.str()); +} diff --git a/src/include/utils/file_access.hpp b/src/include/utils/file_access.hpp new file mode 100644 index 000000000..21fc74c6d --- /dev/null +++ b/src/include/utils/file_access.hpp @@ -0,0 +1,27 @@ +#pragma once + +#include + +#if defined(FLM_CORELIB_TESTING) +#include +#include +#endif + +namespace flm::file_access { + +#if defined(FLM_CORELIB_TESTING) +using OpenObserver = std::function; +inline OpenObserver open_observer; + +inline void SetOpenObserver(OpenObserver observer) { + open_observer = std::move(observer); +} + +inline void ObserveOpen(const std::filesystem::path& path) { + if (open_observer) open_observer(path); +} +#else +inline void ObserveOpen(const std::filesystem::path&) {} +#endif + +} // namespace flm::file_access diff --git a/src/include/utils/npu_platform.hpp b/src/include/utils/npu_platform.hpp new file mode 100644 index 000000000..b00acdcad --- /dev/null +++ b/src/include/utils/npu_platform.hpp @@ -0,0 +1,58 @@ +/// \file npu_platform.hpp +/// \brief the NPU generation this build targets (stx vs aie_next) +/// \note A build carries engines for exactly one generation: the two share +/// none, and FLM_ENABLE_RAI picks which at compile time, so the +/// generation is a build-time fact rather than something to go and ask +/// the hardware. The catalog in model_list.json declares which +/// generations each model supports, and model_list prunes itself to the +/// one this build was made for. Which *kernels* serve a generation is a +/// separate question with its own vocabulary -- see kFlmBackendId and +/// kRaiBackendId in AutoModel/model_backend.hpp. +/// \note aie_next names silicon that has not been announced. When it ships, +/// this enumerator, the string platform_id() returns, and the catalog +/// key in model_list.json are the whole of the rename; nothing on disk +/// and nothing a user has downloaded is named after it. +#pragma once + +#include +#include +#include + +namespace utils { + +/// \brief NPU generation an FLM build targets +enum class npu_platform { + stx, ///< Strix / Krackan Point + aie_next ///< the next NPU generation, not yet announced +}; + +/// \brief the catalog id for a platform, as written in model_list.json +/// \param platform the platform +/// \return "stx" or "aie_next" +constexpr std::string_view platform_id(npu_platform platform) { + return platform == npu_platform::aie_next ? std::string_view("aie_next") + : std::string_view("stx"); +} + +/// \brief the platform every entry is assumed to support when it says nothing +constexpr npu_platform default_npu_platform() { return npu_platform::stx; } + +/// \brief the NPU generation this build has engines for +/// \return aie_next when built with FLM_ENABLE_RAI, stx otherwise +/// \note There is no binary that carries both engines, so this is the whole of +/// platform selection: no probe, and nothing for a user to configure +/// beyond choosing the build that matches their machine. +constexpr npu_platform build_npu_platform() { +#ifdef FLM_ENABLE_RAI + return npu_platform::aie_next; +#else + return npu_platform::stx; +#endif +} + +/// \brief parse a catalog platform id +/// \param text the id, e.g. "aie_next" +/// \return the platform, or nullopt when the id is unknown +std::optional parse_platform(std::string_view text); + +} // namespace utils diff --git a/src/include/utils/utils.hpp b/src/include/utils/utils.hpp index 6766c9cbf..aaa8e66e0 100644 --- a/src/include/utils/utils.hpp +++ b/src/include/utils/utils.hpp @@ -9,10 +9,14 @@ #include "typedef.hpp" #include "buffer.hpp" #include "debug_utils.hpp" +#include "device_runtime.hpp" +#include "nlohmann/json.hpp" #include #include #include #include +#include +#include #ifdef _WIN32 #include #include diff --git a/src/include/utils/vm_args.hpp b/src/include/utils/vm_args.hpp index 1e286b7b5..8d13957d0 100644 --- a/src/include/utils/vm_args.hpp +++ b/src/include/utils/vm_args.hpp @@ -42,6 +42,7 @@ inline void print_help(po::options_description& general) { std::cout << "\tflm pull llama3.2:1b --modelscope 1" << std::endl; std::cout << "\tflm check llama3.2:1b" << std::endl; std::cout << "\tflm serve llama3.2:1b --ctx-len 8192" << std::endl; + std::cout << "\tflm run phi4-mini-it:4b --backend flm_npu" << std::endl; std::cout << "\tflm serve llama3.2:1b --prefill-chunk-len 8192" << std::endl; std::cout << "\tflm serve llama3.2:1b --socket 10" << std::endl; std::cout << "\tflm serve llama3.2:1b --q-len 10" << std::endl; @@ -92,6 +93,8 @@ bool parse_options(int argc, char *argv[], program_args_t& parsed_args) { "Output in JSON format (for list, validate, version commands)") ("ctx-len,c", po::value(&parsed_args.ctx_length)->default_value(-1), "Set context length") + ("backend", po::value(&parsed_args.backend)->default_value(""), + "Execution backend to run the model on (default: the model's own). Overrides FLM_BACKEND") ("prefill-chunk-len,pcl", po::value(&parsed_args.prefill_chunk_len)->default_value(-1), "Set prefill chunk length") ("img-pre-resize,r", po::value(&parsed_args.img_pre_resize)->default_value(2), diff --git a/src/model_info.json b/src/model_info.json index b02ec47c2..923b956ea 100644 --- a/src/model_info.json +++ b/src/model_info.json @@ -3730,5 +3730,31 @@ "size": 165888, "path": "tokenizer_config.json" } + ], + "phi4-mini-it-rai:4b": [ + { + "type": "file", + "path": "Phi-4-mini-instruct.Q8_0.gguf", + "size": 4084611040, + "sha256": "26188c6050d525376a88b04514c236c5e28a36730f1e936f2a00314212b7ba42" + }, + { + "type": "file", + "path": "tokenizer.json", + "size": 15524095, + "sha256": "382cc235b56c725945e149cc25f191da667c836655efd0857b004320e90e91ea" + }, + { + "type": "file", + "path": "tokenizer_config.json", + "size": 2932, + "sha256": "9c9b6bc0c94d95f69f826c41069a3e8b387ac3ced89601d201886e99240ac9db" + }, + { + "type": "file", + "path": "config.json", + "size": 2504, + "sha256": "ac65d86061d3d0d704ee2511fd0eb8713ef19eb6eedba17c3080a4165d5b933b" + } ] } diff --git a/src/model_list.json b/src/model_list.json index cd40dfc7a..92be08133 100644 --- a/src/model_list.json +++ b/src/model_list.json @@ -58,7 +58,7 @@ "parameter_size": "3B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "reasoning" ], "footprint": 3.1 @@ -407,7 +407,7 @@ ], "footprint": 5.0 }, - "9b":{ + "9b": { "name": "Qwen3.5-9B-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3.5-9B-NPU2/resolve/flm_q4k_high_precision", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3.5-9B-NPU2/tree/flm_q4k_high_precision", @@ -440,7 +440,6 @@ "footprint": 8.7 } }, - "qwen3.6-moe": { "35b-a3b": { "name": "Qwen3.6-35B-A3B-NPU2", @@ -611,7 +610,7 @@ "tokenizer_config.json", "chat_template.jinja" ], - "label":[ + "label": [ "reasoning" ], "footprint": 0.96 @@ -641,7 +640,47 @@ "tokenizer.json", "tokenizer_config.json" ], - "footprint": 3.4 + "footprint": 3.4, + "supported_platforms": [ + "stx", + "aie_next" + ], + "platform_overrides": { + "aie_next": { + "name": "phi4-mini-it-rai", + "url": "https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF/resolve/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", + "file_url": "https://huggingface.co/api/models/unsloth/Phi-4-mini-instruct-GGUF/tree/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", + "size": 4100140571, + "default_context_length": 4096, + "details": { + "quantization_level": "Q8_0 -> group-64" + }, + "flm_min_version": "1.0.3", + "files": [ + "Phi-4-mini-instruct.Q8_0.gguf", + "tokenizer.json", + "tokenizer_config.json", + "config.json" + ], + "file_sources": { + "tokenizer.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + }, + "tokenizer_config.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + }, + "config.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + } + }, + "footprint": 4.1, + "ms_url": null, + "model_info_key": "phi4-mini-it-rai:4b" + } + } } }, "embed-gemma": { @@ -666,7 +705,7 @@ "parameter_size": "300M", "quantization_level": "none" }, - "label":[ + "label": [ "embeddings" ], "footprint": 0.62 @@ -695,7 +734,7 @@ "parameter_size": "1B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "audio", "realtime-transcription", "transcription" @@ -754,7 +793,7 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "vision" ], "footprint": 4.5 @@ -786,7 +825,7 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "vision" ], "footprint": 4.5 @@ -818,7 +857,7 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "vision" ], "footprint": 4.5 @@ -850,7 +889,7 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "vision" ], "footprint": 4.5 @@ -955,7 +994,7 @@ "parameter_size": "8B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "reasoning" ], "footprint": 5.4 @@ -985,7 +1024,7 @@ "parameter_size": "8B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "reasoning" ], "footprint": 5.6 @@ -1015,7 +1054,7 @@ "parameter_size": "0.6B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "reasoning" ], "footprint": 0.66 @@ -1043,7 +1082,7 @@ "parameter_size": "1.7B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "reasoning" ], "footprint": 1.6 @@ -1071,7 +1110,7 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "reasoning", "tool-calling" ], @@ -1100,7 +1139,7 @@ "parameter_size": "8B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "reasoning", "tool-calling" ], @@ -1131,7 +1170,7 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "reasoning", "tool-calling" ], @@ -1162,7 +1201,7 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "tool-calling" ], "footprint": 3.1 @@ -1192,7 +1231,7 @@ "parameter_size": "20B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "reasoning" ], "footprint": 14.0 @@ -1222,7 +1261,7 @@ "parameter_size": "20B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "reasoning" ], "footprint": 14.0 @@ -1281,7 +1320,7 @@ "parameter_size": "3B", "quantization_level": "Q4_1" }, - "label":[ + "label": [ "vision" ], "footprint": 3.8 diff --git a/src/pull/download_model.cpp b/src/pull/download_model.cpp index 38ccf44a2..923b8b94f 100644 --- a/src/pull/download_model.cpp +++ b/src/pull/download_model.cpp @@ -15,6 +15,9 @@ #include "nlohmann/json.hpp" #include "picosha2.h" #include "sha1.hpp" +#ifdef _WIN32 +#include +#endif namespace download_utils { @@ -122,6 +125,129 @@ int progress_callback(void* clientp, double dltotal, double dlnow, double ultota return 0; } +namespace { + +FILE* open_part_file(const std::filesystem::path& path, bool append) { +#ifdef _WIN32 + return _wfopen(path.c_str(), append ? L"ab" : L"wb"); +#else + return fopen(path.c_str(), append ? "ab" : "wb"); +#endif +} + +bool promote_atomically(const std::filesystem::path& part, + const std::filesystem::path& destination) { +#ifdef _WIN32 + return MoveFileExW(part.c_str(), destination.c_str(), + MOVEFILE_REPLACE_EXISTING | MOVEFILE_WRITE_THROUGH) != 0; +#else + std::error_code error; + std::filesystem::rename(part, destination, error); + return !error; +#endif +} + +bool request_hash_matches(const DownloadRequest& request, + const std::filesystem::path& path) { + const std::string actual = request.hash_algorithm == HashAlgorithm::Sha256 + ? calculate_file_sha256(path.string()) + : calculate_git_blob_oid(path.string()); + return actual == request.expected_hash; +} + +} // namespace + +bool download_file_atomic(const DownloadRequest& request, + std::function progress_cb) { + if (request.expected_hash.empty()) { + std::cerr << "Missing expected hash for: " << request.destination << std::endl; + return false; + } + + std::error_code error; + std::filesystem::create_directories(request.destination.parent_path(), error); + if (error) { + std::cerr << "Failed to create download directory: " << error.message() << std::endl; + return false; + } + + const std::filesystem::path part(request.destination.string() + ".part"); + std::uint64_t offset = 0; + if (std::filesystem::exists(part, error)) { + offset = std::filesystem::file_size(part, error); + if (error) { + return false; + } + if (offset > request.expected_size) { + std::filesystem::remove(part, error); + if (error) { + return false; + } + offset = 0; + } + } + + if (offset < request.expected_size) { + CURL* curl = curl_easy_init(); + if (!curl) { + std::cerr << "Failed to initialize CURL" << std::endl; + return false; + } + FILE* fp = open_part_file(part, offset != 0); + if (!fp) { + curl_easy_cleanup(curl); + std::cerr << "Failed to open partial file for writing: " << part << std::endl; + return false; + } + + g_progress_bar_shown = false; + hide_cursor(); + curl_easy_setopt(curl, CURLOPT_URL, request.url.c_str()); + curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data_to_file); + curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp); + curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L); + curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); + curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); + curl_easy_setopt(curl, CURLOPT_USERAGENT, "FastFlowLM/1.0"); + curl_easy_setopt(curl, CURLOPT_TIMEOUT, 3600L); + if (offset != 0) { + curl_easy_setopt(curl, CURLOPT_RESUME_FROM_LARGE, + static_cast(offset)); + } + if (progress_cb) { + curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L); + curl_easy_setopt(curl, CURLOPT_PROGRESSFUNCTION, progress_callback); + } + + const CURLcode result = curl_easy_perform(curl); + fclose(fp); + curl_easy_cleanup(curl); + show_cursor(); + if (g_progress_bar_shown) { + std::cout << std::endl; + } + if (result != CURLE_OK) { + std::cerr << "CURL error: " << curl_easy_strerror(result) << std::endl; + return false; // Keep the partial file for the next resume attempt. + } + } + + const std::uint64_t completed_size = std::filesystem::file_size(part, error); + if (error || completed_size != request.expected_size || + !request_hash_matches(request, part)) { + std::filesystem::remove(part, error); + header_print("FLM", "Downloaded file size or hash did not match."); + return false; + } + + if (!promote_atomically(part, request.destination)) { + std::cerr << "Failed to atomically promote: " << request.destination << std::endl; + return false; + } + header_print("FLM", "Download completed: " << request.destination.string()); + return true; +} + /// \brief Download a file from URL to a local file /// \param url the URL to download from /// \param local_path the local path to save the file @@ -218,6 +344,22 @@ static bool download_with_retry(const std::string& url, const std::string& local return false; } +static bool download_with_retry(const DownloadRequest& request, + std::function progress_cb, + int max_retries = 3) { + for (int attempt = 0; attempt < max_retries; ++attempt) { + if (download_file_atomic(request, progress_cb)) { + return true; + } + header_print("FLM", "Download failed (attempt " << (attempt + 1) << "/" << max_retries << ")"); + if (attempt + 1 < max_retries) { + header_print("FLM", "Retrying..."); + std::this_thread::sleep_for(std::chrono::seconds(1)); + } + } + return false; +} + /// \brief Download content from URL to a string /// \param url the URL to download from /// \return the downloaded string @@ -268,6 +410,14 @@ bool download_multiple_files(const nlohmann::json downloads, std::string filename = std::filesystem::path(url).filename().string(); std::string remote_oid = file["oid"]; bool is_lfs = file["is_lfs"]; + DownloadRequest request{ + url, + local_path, + file["expected_size"].get(), + file.value("hash_algorithm", std::string()) == "sha256" + ? HashAlgorithm::Sha256 + : HashAlgorithm::GitBlobSha1, + remote_oid}; // cut "?download=true" if (filename.find("?download=true") != std::string::npos) { @@ -282,7 +432,7 @@ bool download_multiple_files(const nlohmann::json downloads, } }; - if (!download_with_retry(url, local_path, is_lfs, remote_oid, file_progress)) { + if (!download_with_retry(request, file_progress)) { std::cerr << "Failed to download: " << url << std::endl; //show_cursor(); // Show cursor on error return false; diff --git a/src/pull/download_model.hpp b/src/pull/download_model.hpp index 12e9a1a19..d160bf9d3 100644 --- a/src/pull/download_model.hpp +++ b/src/pull/download_model.hpp @@ -7,6 +7,8 @@ #pragma once #include +#include +#include #include #include #include @@ -15,6 +17,16 @@ namespace download_utils { +enum class HashAlgorithm { Sha256, GitBlobSha1 }; + +struct DownloadRequest { + std::string url; + std::filesystem::path destination; + std::uint64_t expected_size; + HashAlgorithm hash_algorithm; + std::string expected_hash; +}; + std::string calculate_file_sha256(const std::string& file_path); std::string calculate_git_blob_oid(const std::string& file_path); @@ -35,6 +47,11 @@ int progress_callback(void* clientp, double dltotal, double dlnow, double ultota bool download_file(const std::string& url, const std::string& local_path, bool is_lfs, std::string remote_oid, std::function progress_cb = nullptr); +// Download to a same-directory temporary file, verify it, then atomically promote it. +bool download_file_atomic( + const DownloadRequest& request, + std::function progress_cb = nullptr); + // Download content from URL to a string std::string download_string(const std::string& url); diff --git a/src/pull/model_downloader.cpp b/src/pull/model_downloader.cpp index 24fc98d4f..d9256cba0 100644 --- a/src/pull/model_downloader.cpp +++ b/src/pull/model_downloader.cpp @@ -10,6 +10,157 @@ #include #include #include +#include +#include + +namespace { + +std::string percent_encode_filename(std::string_view filename) { + static constexpr char kHex[] = "0123456789ABCDEF"; + std::string encoded; + for (const unsigned char ch : filename) { + if ((ch >= 'a' && ch <= 'z') || (ch >= 'A' && ch <= 'Z') || + (ch >= '0' && ch <= '9') || ch == '-' || ch == '_' || ch == '.' || ch == '~') { + encoded.push_back(static_cast(ch)); + } else { + encoded.push_back('%'); + encoded.push_back(kHex[ch >> 4]); + encoded.push_back(kHex[ch & 0x0f]); + } + } + return encoded; +} + +bool is_hex_revision(const std::string& revision) { + return revision.size() == 40 && + std::all_of(revision.begin(), revision.end(), [](unsigned char ch) { + return std::isxdigit(ch) != 0; + }); +} + +/// \brief load the pinned per-file records describing a model's download +/// \param model_info the resolved catalog entry +/// \param model_tag the resolved "family:size" tag +/// \note A model whose artifacts differ per NPU generation shares one tag across +/// platforms, so its entry names the record set explicitly via +/// "model_info_key"; everything else is keyed by its tag. +nlohmann::json load_model_file_records(const nlohmann::json& model_info, + const std::string& model_tag) { + const std::string key = model_info.value("model_info_key", model_tag); + std::ifstream stream(utils::find_model_info()); + if (!stream.is_open()) { + throw std::runtime_error("model_info.json could not be opened"); + } + return nlohmann::json::parse(stream).at(key); +} + +const nlohmann::json& find_file_record(const nlohmann::json& records, + const std::string& filename) { + const auto record = std::find_if(records.begin(), records.end(), [&](const auto& value) { + return value.at("path") == filename; + }); + if (record == records.end()) { + throw std::runtime_error("missing model_info record for " + filename); + } + return *record; +} + +struct ResolvedModelFile { + ModelFileSource source; + std::uint64_t size; + bool is_lfs; + download_utils::HashAlgorithm hash_algorithm; + std::string hash; +}; + +/// \brief whether an entry's files come from pinned upstream revisions +/// \param model_info the resolved model_list.json entry +/// \return true when the entry carries a "file_sources" map +/// \note These entries are assembled from third-party repos at a fixed commit +/// rather than published by FastFlowLM, so every file is hash-verified +/// against the same pinned metadata it was downloaded with. The rai +/// Phi-4 GGUF is the only such entry today, but the rule is about how the +/// files are sourced, not about which backend consumes them. +bool uses_pinned_upstream_integrity(const nlohmann::json& model_info) { + // Presence, not contents: resolve_file_source decides the same way, and a + // malformed map should fail there rather than quietly downgrade an entry to + // the unverified path here. + return model_info.contains("file_sources"); +} + +} // namespace + +ModelFileSource resolve_file_source(const nlohmann::json& model_info, + std::string_view filename, + bool use_modelscope) { + if (model_info.contains("file_sources")) { + if (use_modelscope) { + throw std::runtime_error("pinned Hugging Face per-file sources are required; --modelscope is not supported"); + } + const auto& sources = model_info.at("file_sources"); + if (!sources.is_object()) { + throw std::runtime_error("file_sources must be an object"); + } + std::unordered_set files; + for (const auto& file : model_info.at("files")) { + files.insert(file.get()); + } + for (const auto& [key, value] : sources.items()) { + if (!files.contains(key)) { + throw std::runtime_error("unknown file_sources key: " + key); + } + if (!value.is_object() || value.size() != 2 || + !value.contains("url") || !value.at("url").is_string() || + value.at("url").get().empty()) { + throw std::runtime_error("file source requires exactly a non-empty string url and revision"); + } + if (!value.contains("revision") || !value.at("revision").is_string() || + !is_hex_revision(value.at("revision").get())) { + throw std::runtime_error("file source revision must be a 40-character hexadecimal string"); + } + } + const auto override = sources.find(std::string(filename)); + if (override != sources.end()) { + const std::string base = override->at("url"); + const std::string revision = override->at("revision"); + return {base + "/resolve/" + revision + "/" + + percent_encode_filename(filename) + "?download=true", + revision}; + } + } + + const std::string base_url = use_modelscope + ? model_info.at("ms_url").get() + : model_info.at("url").get(); + if (base_url.find("resolve") != std::string::npos) { + return {base_url + "/" + std::string(filename) + "?download=true", {}}; + } + return {base_url + "/resolve/main/" + std::string(filename) + "?download=true", {}}; +} + +namespace { + +ResolvedModelFile resolve_model_file(const nlohmann::json& model_info, + const nlohmann::json& records, + const std::string& filename, + bool use_modelscope) { + const auto& record = find_file_record(records, filename); + const bool is_lfs = record.contains("lfs"); + const bool has_explicit_sha256 = record.contains("sha256"); + return { + resolve_file_source(model_info, filename, use_modelscope), + record.at("size").get(), + is_lfs, + has_explicit_sha256 || is_lfs + ? download_utils::HashAlgorithm::Sha256 + : download_utils::HashAlgorithm::GitBlobSha1, + has_explicit_sha256 + ? record.at("sha256").get() + : (is_lfs ? record.at("lfs").at("oid").get() + : record.at("oid").get())}; +} + +} // namespace /// \brief Constructor /// \param models the model list @@ -22,23 +173,27 @@ ModelDownloader::ModelDownloader(model_list& models) /// \param model_tag the model tag /// \return true if the model is downloaded, false otherwise ModelDownloader::ModelStatus ModelDownloader::is_model_downloaded(const std::string& model_tag, bool sub_process_mode, bool fast_check) { - auto missing_files = get_missing_files(model_tag); + const auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); + const bool strict_integrity = uses_pinned_upstream_integrity(model_info); + auto missing_files = get_missing_files(new_model_tag); bool is_config_file_missing = std::find(missing_files.begin(), missing_files.end(), "config.json") != missing_files.end(); ModelStatus modelstatus = ModelStatus::Missing; if (!is_config_file_missing) { - modelstatus = check_model_compatibility(model_tag, sub_process_mode); + modelstatus = check_model_compatibility(new_model_tag, sub_process_mode); if (modelstatus == ModelStatus::Outdated) { if (!fast_check) { header_print("FLM", "Checking outdated files..."); - verify_and_clean_files(model_tag, sub_process_mode); + verify_and_clean_files(new_model_tag, false, sub_process_mode); } } - else if (modelstatus == ModelStatus::Ready && !missing_files.empty()) { - // config.json is present and the version check passed, but other - // files (e.g. weights) are still missing. - modelstatus = ModelStatus::Missing; + else if (modelstatus == ModelStatus::Ready) { + if (!missing_files.empty() || + (strict_integrity && !fast_check && + !verify_and_clean_files(new_model_tag, false, sub_process_mode))) { + modelstatus = ModelStatus::Missing; + } } } return modelstatus; @@ -51,8 +206,12 @@ ModelDownloader::ModelStatus ModelDownloader::check_model_compatibility(const st auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); LM_Config config; config.from_pretrained(this->supported_models.get_model_path(new_model_tag)); - std::string flm_version = config.flm_version; std::string flm_min_version = model_info["flm_min_version"]; + // The pinned Microsoft frontend config is upstream-native and intentionally + // has no FLM version. Its catalog contract supplies the compatibility floor. + std::string flm_version = uses_pinned_upstream_integrity(model_info) + ? flm_min_version + : config.flm_version; int l_l, m_l, r_l; //left, middle, right on local version int l_r, m_r, r_r; //left, middle, right on requried version int l_f, m_f, r_f; //left, middle, right on flm version @@ -89,6 +248,10 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); std::string model_name = model_info["name"]; std::string model_server = use_modelscope ? "ModelScope" : "HuggingFace"; + if (use_modelscope && model_info.contains("file_sources")) { + // Validate this before any ready-state early return. + resolve_file_source(model_info, model_info.at("files").at(0).get(), true); + } header_print("FLM", "Pulling model from " + model_server + "..."); header_print("FLM", "Model: " + new_model_tag); @@ -101,9 +264,13 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco header_print("FLM", "Model already downloaded. Use --force to re-download."); return true; } - verify_and_clean_files(new_model_tag, use_modelscope); break; case ModelStatus::Missing: + if (uses_pinned_upstream_integrity(model_info)) { + // Preserve valid finals, but remove corrupt pinned finals before + // deciding which files need to be downloaded. + verify_and_clean_files(new_model_tag, use_modelscope, true); + } break; case ModelStatus::Outdated: break; @@ -137,12 +304,13 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco } // Build download list - auto download_list = build_download_list(new_model_tag, use_modelscope); + auto download_list = build_download_list(new_model_tag, use_modelscope, force_redownload); auto downloads = download_list.first; float sum_fize_size = download_list.second; if (downloads.empty()) { header_print("FLM", "No files to download for model: " + new_model_tag); - return true; // Return true since all files are already present + return !uses_pinned_upstream_integrity(model_info) || + verify_and_clean_files(new_model_tag, use_modelscope); } header_print("FLM", "Downloading " + std::to_string(downloads.size()) + " missing files..."); @@ -162,17 +330,17 @@ bool ModelDownloader::pull_model(const std::string& model_tag, bool use_modelsco if (success) { header_print("FLM", "Model downloaded successfully!"); - // Verify download + // Verify every final file using the same pinned metadata used to download it. auto final_missing = get_missing_files(new_model_tag); - if (final_missing.empty()) { + const bool verified = final_missing.empty() && + (!uses_pinned_upstream_integrity(model_info) || + verify_and_clean_files(new_model_tag, use_modelscope)); + if (verified) { header_print("FLM", "All files verified successfully."); } else { - header_print("WARNING", "Some files may be missing after download:"); - for (const auto& file : final_missing) { - std::cout << " - " << file << std::endl; - } + header_print("WARNING", "Some files are missing or failed verification after download."); } - return true; + return verified; } else { header_print("ERROR", "Failed to download model files."); return false; @@ -285,82 +453,39 @@ std::string ModelDownloader::get_model_file_path(const std::string& model_path, /// \brief Build the download list /// \param model_tag the model tag /// \return the download list -std::pair ModelDownloader::build_download_list(const std::string& model_tag, bool modelscope) { - +std::pair ModelDownloader::build_download_list( + const std::string& model_tag, bool modelscope, bool force_redownload) { nlohmann::json downloads = nlohmann::json::array(); float sum_file_size = 0; - try { - auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); - std::string base_url = modelscope ? model_info["ms_url"] : model_info["url"]; - std::string model_name = model_info["name"]; - std::string file_url = model_info["file_url"]; - std::vector model_files = model_info["files"]; - - // Create model directory - std::string model_path = supported_models.get_model_path(new_model_tag); - std::filesystem::create_directories(model_path); - - nlohmann::json hf_model_infos; - // GET HF api/models - // if (modelscope == 0) { - // std::string hf_response = download_utils::download_string(file_url); - // hf_model_infos = nlohmann::json::parse(hf_response); - // } - // else { - std::string model_info_path = utils::find_model_info(); - std::ifstream model_info_file(model_info_path); - nlohmann::json model_info_json = nlohmann::json::parse(model_info_file); - hf_model_infos = model_info_json.at(new_model_tag); - // } - - for (const auto& filename : model_files) { - auto it = std::find_if( - hf_model_infos.begin(), - hf_model_infos.end(), - [&](const nlohmann::json& f) { - return f["path"] == filename; - } - ); - if (it == hf_model_infos.end()) { - continue; - } - - const auto& file = *it; - std::string local_path = get_model_file_path(model_path, filename); - - if (!file_exists(local_path)) { - std::string url; - if (std::string(base_url).find("resolve") != std::string::npos) { // resolve provided , may from a specific branch - url = base_url + "/" + filename + "?download=true"; - } - else { - url = base_url + "/resolve/main/" + filename + "?download=true"; - } - // header_print("URL", url); - bool is_lfs = file.contains("lfs"); - std::string oid = is_lfs ? file["lfs"]["oid"] : file["oid"]; - float file_size = static_cast(file["size"]) / 1024 / 1024; - sum_file_size += file_size; - - nlohmann::json entry = { - {"file", filename}, - {"size", file_size}, - {"url", url}, - {"localpath", local_path}, - {"oid", oid}, - {"is_lfs", is_lfs}, - }; - downloads.push_back(entry); - } - + auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); + const std::vector model_files = model_info.at("files"); + const std::string model_path = supported_models.get_model_path(new_model_tag); + std::filesystem::create_directories(model_path); + const nlohmann::json records = load_model_file_records(model_info, new_model_tag); + + for (const auto& filename : model_files) { + const std::string local_path = get_model_file_path(model_path, filename); + if (!force_redownload && file_exists(local_path)) { + continue; } - } - catch (const std::exception& e) { - header_print("ERROR", "Error building download list: " + std::string(e.what())); - } - return std::make_pair(downloads, sum_file_size); + const auto file = resolve_model_file(model_info, records, filename, modelscope); + const float file_size = static_cast(file.size) / 1024 / 1024; + sum_file_size += file_size; + downloads.push_back({ + {"file", filename}, + {"size", file_size}, + {"expected_size", file.size}, + {"url", file.source.url}, + {"localpath", local_path}, + {"oid", file.hash}, + {"is_lfs", file.is_lfs}, + {"hash_algorithm", file.hash_algorithm == download_utils::HashAlgorithm::Sha256 + ? "sha256" : "git_blob_sha1"}, + }); + } + return {downloads, sum_file_size}; } /// \brief Remove a model and all its files @@ -421,18 +546,30 @@ bool ModelDownloader::remove_model(const std::string& model_tag, bool sub_proces /// \return true if all files are present and compatible, false otherwise bool ModelDownloader::check_model(const std::string& model_tag, bool use_modelscope, bool sub_process_mode) { auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); + if (use_modelscope && model_info.contains("file_sources")) { + try { + resolve_file_source( + model_info, model_info.at("files").at(0).get(), true); + } + catch (const std::exception& error) { + header_print("ERROR", error.what()); + return false; + } + } header_print("FLM", "Checking model: " + new_model_tag + "...\n"); - ModelStatus status = is_model_downloaded(new_model_tag, sub_process_mode); + // check_model owns the one full integrity pass below. Status discovery must + // remain presence/version-only so a pinned 4.1 GB model is not hashed twice. + ModelStatus status = is_model_downloaded(new_model_tag, sub_process_mode, true); switch (status) { case ModelStatus::Missing: header_print("FLM", "Model not found: " + new_model_tag); header_print("FLM", "Use `flm pull " + new_model_tag + "` to download it."); - return true; + return false; case ModelStatus::Incompatible: header_print("FLM", "Model is incompatible with this version of FastFlowLM: " + new_model_tag); header_print("FLM", "Use `flm pull " + new_model_tag + "` to re-download it."); - return true; + return false; case ModelStatus::Outdated: case ModelStatus::Ready: { bool ok = verify_and_clean_files(new_model_tag, use_modelscope, sub_process_mode); @@ -440,10 +577,10 @@ bool ModelDownloader::check_model(const std::string& model_tag, bool use_modelsc header_print("FLM", "Model check completed with errors. Use `flm pull " + new_model_tag + "` to re-download corrupted files."); else header_print("FLM", "Model check completed successfully. All files are present and compatible."); - return true; + return ok; } } - return true; + return false; } /// \brief Verify each model file's hash against HuggingFace metadata and @@ -457,37 +594,15 @@ bool ModelDownloader::verify_and_clean_files(const std::string& model_tag, bool auto [new_model_tag, model_info] = supported_models.get_model_info(model_tag); std::vector model_files = model_info["files"]; std::string model_path = supported_models.get_model_path(new_model_tag); - std::string file_url = model_info["file_url"]; - - nlohmann::json hf_model_infos; - // GET HF api/models - // if (use_modelscope == 0) { - // std::string hf_response = download_utils::download_string(file_url); - // hf_model_infos = nlohmann::json::parse(hf_response); - // } - // else { - std::string model_info_path = utils::find_model_info(); - std::ifstream model_info_file(model_info_path); - nlohmann::json model_info_json = nlohmann::json::parse(model_info_file); - hf_model_infos = model_info_json.at(new_model_tag); - // } + const nlohmann::json records = load_model_file_records(model_info, new_model_tag); for (const auto& filename : model_files) { if (!sub_process_mode) { header_print("FLM", "Checking file: " + filename + "..."); } - auto it = std::find_if( - hf_model_infos.begin(), - hf_model_infos.end(), - [&](const nlohmann::json& f) { - return f["path"] == filename; - } - ); - if (it == hf_model_infos.end()) { - continue; - } - const auto& file = *it; + const auto file = resolve_model_file( + model_info, records, filename, use_modelscope); std::string local_path = get_model_file_path(model_path, filename); // If the file isn't present locally, there's nothing to verify or @@ -498,11 +613,15 @@ bool ModelDownloader::verify_and_clean_files(const std::string& model_tag, bool continue; } - bool is_lfs = file.contains("lfs"); - std::string oid_ref = is_lfs ? file["lfs"]["oid"] : file["oid"]; - std::string local_oid = is_lfs ? download_utils::calculate_file_sha256(local_path) : download_utils::calculate_git_blob_oid(local_path); + const std::string local_oid = + file.hash_algorithm == download_utils::HashAlgorithm::Sha256 + ? download_utils::calculate_file_sha256(local_path) + : download_utils::calculate_git_blob_oid(local_path); + std::error_code size_error; + const auto local_size = std::filesystem::file_size(local_path, size_error); + const bool size_matches = !size_error && local_size == file.size; - if (local_oid == oid_ref) { + if (size_matches && local_oid == file.hash) { if (!sub_process_mode) { header_print("FLM", "Success!"); } diff --git a/src/pull/model_downloader.hpp b/src/pull/model_downloader.hpp index 52a90f813..4c1bd1fba 100644 --- a/src/pull/model_downloader.hpp +++ b/src/pull/model_downloader.hpp @@ -14,6 +14,17 @@ #include #include #include +#include + +struct ModelFileSource { + std::string url; + std::string revision; +}; + +ModelFileSource resolve_file_source( + const nlohmann::json& model_info, + std::string_view filename, + bool use_modelscope); class ModelDownloader { public: @@ -59,7 +70,8 @@ class ModelDownloader { std::string get_model_file_path(const std::string& model_path, const std::string& filename); // Build download URLs for model files - std::pair build_download_list(const std::string& model_tag, bool modelscope=0); + std::pair build_download_list( + const std::string& model_tag, bool modelscope=0, bool force_redownload=false); // bool check_model_compatibility(const std::string& model_tag); ModelStatus check_model_compatibility(const std::string& model_tag, bool sub_process_mode=0); diff --git a/src/runner/runner.cpp b/src/runner/runner.cpp index d38e5e9c1..c36c7fcc5 100644 --- a/src/runner/runner.cpp +++ b/src/runner/runner.cpp @@ -38,10 +38,9 @@ std::map cmd_map = { /// \param supported_models - the list of supported models /// \param downloader - the downloader for the models /// \param tag - the tag of the model to load -Runner::Runner(model_list& supported_models, ModelDownloader& downloader, program_args_t& args) - : supported_models(supported_models), downloader(downloader), tag(args.model_tag), modelscope(args.modelscope), asr(args.asr), embed(args.embed), img_pre_resize(args.img_pre_resize), preemption(args.preemption) { - - this->npu_device_inst = flm_rt::device(0); +Runner::Runner(model_list& supported_models, ModelDownloader& downloader, program_args_t& args, + flm_rt::device* npu_device) + : supported_models(supported_models), downloader(downloader), tag(args.model_tag), modelscope(args.modelscope), asr(args.asr), embed(args.embed), img_pre_resize(args.img_pre_resize), preemption(args.preemption), backend(args.backend), npu_device_inst(npu_device) { if (args.ctx_length != -1) { this->ctx_length = args.ctx_length >= 512 ? args.ctx_length : 512; @@ -53,12 +52,12 @@ Runner::Runner(model_list& supported_models, ModelDownloader& downloader, progra if (this->auto_chat_engine != nullptr) { this->auto_chat_engine.reset(); } - std::pair> auto_model = get_auto_model(this->tag, this->supported_models, &this->npu_device_inst); + std::pair> auto_model = get_auto_model(this->tag, this->supported_models, this->npu_device_inst); this->auto_chat_engine = std::move(auto_model.second); this->tag = auto_model.first; - switch (this->downloader.is_model_downloaded(this->tag)) { + switch (this->downloader.is_model_downloaded(this->tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: @@ -73,7 +72,9 @@ Runner::Runner(model_list& supported_models, ModelDownloader& downloader, progra // header_print("ASR", asr_supported); this->auto_chat_engine->configure_parameter("img_pre_resize", this->img_pre_resize); try { - this->auto_chat_engine->load_model(this->supported_models.get_model_path(new_tag), model_info, this->ctx_length, this->preemption); + const auto load_started = std::chrono::steady_clock::now(); + this->auto_chat_engine->load_model(this->supported_models.get_model_path(new_tag), model_info, this->ctx_length, this->preemption, this->backend); + report_load_time(load_started); } catch (const std::exception& e) { header_print("ERROR", "Failed to load model: " + std::string(e.what())); @@ -119,7 +120,7 @@ Runner::Runner(model_list& supported_models, ModelDownloader& downloader, progra header_print("ERROR", "Whisper is incompatible with this version of FastFlowLM, skipping... "); return; } - this->whisper_engine = std::make_unique(&this->npu_device_inst); + this->whisper_engine = std::make_unique(this->npu_device_inst); auto [new_whisper_tag, whisper_model_info] = this->supported_models.get_model_info(whisper_tag); std::string whisper_model_path = this->supported_models.get_model_path(new_whisper_tag); try { @@ -358,6 +359,8 @@ void Runner::run() { chat_meta_info_t meta_info; meta_info.max_prefill_len = this->prefill_chunk_len; uniformed_input.prompt = input; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens(this->generate_limit); this->auto_chat_engine->start_total_timer(); @@ -425,13 +428,13 @@ void Runner::cmd_status(std::vector& input_list) { void Runner::cmd_load(std::vector& input_list) { std::string model_name = input_list[1]; - std::pair> auto_model = get_auto_model(model_name, this->supported_models, &this->npu_device_inst); + std::pair> auto_model = get_auto_model(model_name, this->supported_models, this->npu_device_inst); model_name = auto_model.first; if (model_name != this->tag) { this->tag = model_name; - switch (this->downloader.is_model_downloaded(this->tag)) { + switch (this->downloader.is_model_downloaded(this->tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: @@ -450,7 +453,9 @@ void Runner::cmd_load(std::vector& input_list) { auto [new_tag, model_info] = this->supported_models.get_model_info(this->tag); this->auto_chat_engine->configure_parameter("img_pre_resize", this->img_pre_resize); try { - this->auto_chat_engine->load_model(this->supported_models.get_model_path(new_tag), model_info, this->ctx_length, this->preemption); + const auto load_started = std::chrono::steady_clock::now(); + this->auto_chat_engine->load_model(this->supported_models.get_model_path(new_tag), model_info, this->ctx_length, this->preemption, this->backend); + report_load_time(load_started); } catch (const std::exception& e) { header_print("ERROR", "Failed to load model: " + std::string(e.what())); diff --git a/src/runner/runner.hpp b/src/runner/runner.hpp index 889c5bacc..7c21225e9 100644 --- a/src/runner/runner.hpp +++ b/src/runner/runner.hpp @@ -44,7 +44,8 @@ typedef enum { /// \brief Runner class class Runner { public: - Runner(model_list& supported_models, ModelDownloader& downloader, program_args_t& args); + Runner(model_list& supported_models, ModelDownloader& downloader, program_args_t& args, + flm_rt::device* npu_device); void run(); private: std::string tag; @@ -64,10 +65,15 @@ class Runner { int ctx_length; std::string system_prompt; bool preemption; + /// \brief the --backend value, empty when the flag was not given + std::string backend; int img_pre_resize; // CLI instance for interactive input CLIWide cli; - flm_rt::device npu_device_inst; + // Owned by main(); on a rai build it comes from corelib, on an flm + // build main opens it directly. Either way there is one per process and + // it must not be duplicated. + flm_rt::device* npu_device_inst; /// \brief Command functions void cmd_set(std::vector& input_list); diff --git a/src/server/rest_handler.cpp b/src/server/rest_handler.cpp index 7890a4b5b..55a864e45 100644 --- a/src/server/rest_handler.cpp +++ b/src/server/rest_handler.cpp @@ -20,6 +20,21 @@ #include #include "server.hpp" +namespace { +json ModelErrorResponse(const ModelRequestError& error) { + return {{"error", {{"message", error.what()}, + {"type", "model_error"}, + {"code", error.http_code()}, + {"session_cleared", error.session_cleared()}}}}; +} + +json ExceptionResponse(const std::exception& error) { + if (const auto* model_error = dynamic_cast(&error)) + return ModelErrorResponse(*model_error); + return {{"error", error.what()}}; +} +} + ///@brief Normalize messages by merging consecutive user messages (like Ollama does) ///@param messages the original messages ///@return normalized messages with consecutive user messages merged @@ -321,9 +336,9 @@ static json convert_tool_responses_gemma4(json messages) { ///@param embed whether to enable embedding ///@return the rest handler -RestHandler::RestHandler(model_list& models, ModelDownloader& downloader, program_args_t& args) - : supported_models(models), downloader(downloader), default_model_tag(args.model_tag), current_model_tag(""), modelscope(args.modelscope), asr(args.asr), embed(args.embed), img_pre_resize(args.img_pre_resize), preemption(args.preemption){ - this->npu_device_inst = flm_rt::device(0); +RestHandler::RestHandler(model_list& models, ModelDownloader& downloader, program_args_t& args, + flm_rt::device* npu_device) + : supported_models(models), downloader(downloader), default_model_tag(args.model_tag), current_model_tag(""), modelscope(args.modelscope), asr(args.asr), embed(args.embed), img_pre_resize(args.img_pre_resize), preemption(args.preemption), backend(args.backend), npu_device_inst(npu_device){ if (args.ctx_length != -1) { this->ctx_length = args.ctx_length >= 512 ? args.ctx_length : 512; @@ -377,18 +392,23 @@ RestHandler::~RestHandler() = default; ///@brief Ensure the model is loaded ///@param model_tag the model tag -bool RestHandler::ensure_model_loaded(const std::string& model_tag) { +bool RestHandler::ensure_model_loaded(const std::string& model_tag, + const std::string& request_backend) { + // A per-request "backend" overrides --backend; either one differing from + // what is loaded forces a reload, exactly as a different model tag does. + const std::string requested_backend = + request_backend.empty() ? this->backend : request_backend; std::string ensure_tag = model_tag; - if (current_model_tag != ensure_tag) { + if (current_model_tag != ensure_tag || current_backend != requested_backend) { std::this_thread::sleep_for(std::chrono::milliseconds(500)); if (auto_chat_engine != nullptr) { auto_chat_engine.reset(); } - std::pair> auto_model = get_auto_model(ensure_tag, this->supported_models, &this->npu_device_inst); + std::pair> auto_model = get_auto_model(ensure_tag, this->supported_models, this->npu_device_inst); auto_chat_engine = std::move(auto_model.second); auto_chat_engine->set_server_mode(true); ensure_tag = auto_model.first; - switch (downloader.is_model_downloaded(ensure_tag)) { + switch (downloader.is_model_downloaded(ensure_tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: @@ -401,13 +421,16 @@ bool RestHandler::ensure_model_loaded(const std::string& model_tag) { auto [new_ensure_tag, model_info] = supported_models.get_model_info(ensure_tag); auto_chat_engine->configure_parameter("img_pre_resize", this->img_pre_resize); try { - auto_chat_engine->load_model(supported_models.get_model_path(new_ensure_tag), model_info, ctx_length, preemption); + const auto load_started = std::chrono::steady_clock::now(); + auto_chat_engine->load_model(supported_models.get_model_path(new_ensure_tag), model_info, ctx_length, preemption, requested_backend); + report_load_time(load_started); } catch (const std::exception& e) { header_print("ERROR", "Failed to load model: " + std::string(e.what())); this->auto_chat_engine.reset(); - this->npu_device_inst.reset(); - this->npu_device_inst = flm_rt::device(0); + // The device is owned by main() and shared with every engine, so it + // is deliberately left alone here; releasing the failed engine is + // what frees the hardware context. this->current_model_tag = "model-faker"; return false; } @@ -416,6 +439,7 @@ bool RestHandler::ensure_model_loaded(const std::string& model_tag) { this->prefill_chunk_len = model_info["max_prefill_len"].get();; } current_model_tag = ensure_tag; + current_backend = requested_backend; } return true; } @@ -425,7 +449,7 @@ bool RestHandler::ensure_model_loaded(const std::string& model_tag) { void RestHandler::ensure_asr_model_loaded(const std::string& model_tag) { #ifndef FASTFLOWLM_LINUX_LIMITED_MODELS std::string ensure_tag = model_tag; - switch (downloader.is_model_downloaded(ensure_tag)) { + switch (downloader.is_model_downloaded(ensure_tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: @@ -437,7 +461,7 @@ void RestHandler::ensure_asr_model_loaded(const std::string& model_tag) { this->asr = false; return; } - this->whisper_engine = std::make_unique(&this->npu_device_inst); + this->whisper_engine = std::make_unique(this->npu_device_inst); auto [new_ensure_tag, whisper_model_info] = this->supported_models.get_model_info(ensure_tag); std::string whisper_model_path = this->supported_models.get_model_path(new_ensure_tag); try { @@ -457,7 +481,7 @@ void RestHandler::ensure_asr_model_loaded(const std::string& model_tag) { void RestHandler::ensure_embed_model_loaded(const std::string& model_tag) { #ifndef FASTFLOWLM_LINUX_LIMITED_MODELS std::string ensure_tag = model_tag; - switch (this->downloader.is_model_downloaded(ensure_tag)) { + switch (this->downloader.is_model_downloaded(ensure_tag, false, /*fast_check=*/true)) { case ModelDownloader::ModelStatus::Ready: break; case ModelDownloader::ModelStatus::Outdated: @@ -469,7 +493,7 @@ void RestHandler::ensure_embed_model_loaded(const std::string& model_tag) { this->embed = false; return; } - auto [embedding_model_tag, auto_embedding_engine] = get_auto_embedding_model(ensure_tag, &this->npu_device_inst); + auto [embedding_model_tag, auto_embedding_engine] = get_auto_embedding_model(ensure_tag, this->npu_device_inst); this->auto_embedding_engine = std::move(auto_embedding_engine); auto [new_embedding_model_tag, embedding_model_info] = this->supported_models.get_model_info(embedding_model_tag); std::string embedding_model_path = this->supported_models.get_model_path(new_embedding_model_tag); @@ -650,12 +674,13 @@ void RestHandler::handle_generate(const json& request, std::string prompt = request["prompt"]; bool stream = request.value("stream", true); std::string model = request.value("model", current_model_tag); + const std::string request_backend = request.value("backend", std::string()); json options = request.value("options", json::object()); int length_limit = request.value("max_tokens", 4096); auto load_start_time = time_utils::now(); // TODO: Use Another Check Function avoid loading again - if (!ensure_model_loaded(model)) { + if (!ensure_model_loaded(model, request_backend)) { json error_response = {{"error", "Failed to load " + model + " model!"}}; send_response(error_response); return; @@ -664,6 +689,11 @@ void RestHandler::handle_generate(const json& request, chat_meta_info_t meta_info; lm_uniform_input_t uniformed_input; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens( + request.contains("max_tokens") + ? std::optional(request.at("max_tokens").get()) + : std::nullopt); meta_info.max_prefill_len = this->prefill_chunk_len; meta_info.load_duration = (uint64_t)time_utils::duration_ns(load_start_time, load_end_time).first; header_print("FLM", "Start generating..."); @@ -674,7 +704,7 @@ void RestHandler::handle_generate(const json& request, streaming_ostream ostream(model, send_streaming_response, false); uniformed_input.prompt = prompt; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success){ json error_response = {{"error", "Max length reached"}}; send_response(error_response); @@ -682,15 +712,15 @@ void RestHandler::handle_generate(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - auto_chat_engine->generate(meta_info, length_limit, ostream); + auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -707,7 +737,7 @@ void RestHandler::handle_generate(const json& request, std::ostream ostream(&obuf); uniformed_input.prompt = prompt; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success){ json error_response = {{"error", "Max length reached"}}; send_response(error_response); @@ -715,15 +745,15 @@ void RestHandler::handle_generate(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - auto_chat_engine->generate(meta_info, length_limit, ostream); + auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -747,7 +777,7 @@ void RestHandler::handle_generate(const json& request, send_response(response); } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -764,11 +794,12 @@ void RestHandler::handle_chat(const json& request, nlohmann::ordered_json messages = request["messages"]; bool stream = request.value("stream", false); std::string model = request.value("model", current_model_tag); + const std::string request_backend = request.value("backend", std::string()); json options = request.value("options", json::object()); int length_limit = options.value("num_predict", 4096); auto load_start_time = time_utils::now(); - if (!ensure_model_loaded(model)) { + if (!ensure_model_loaded(model, request_backend)) { json error_response = {{"error", "Failed to load " + model + " model!"}}; send_response(error_response); return; @@ -781,6 +812,11 @@ void RestHandler::handle_chat(const json& request, chat_meta_info_t meta_info; lm_uniform_input_t uniformed_input; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens( + options.contains("num_predict") + ? std::optional(options.at("num_predict").get()) + : std::nullopt); meta_info.load_duration = (uint64_t)time_utils::duration_ns(load_start_time, load_end_time).first; meta_info.max_prefill_len = this->prefill_chunk_len; header_print("FLM", "Start generating..."); @@ -790,7 +826,7 @@ void RestHandler::handle_chat(const json& request, streaming_ostream ostream(model, send_streaming_response, true); // true for chat format uniformed_input.messages = messages; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success){ json error_response = {{"error", "Max length reached"}}; send_response(error_response); @@ -798,21 +834,16 @@ void RestHandler::handle_chat(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); - if (!success){ - json error_response = {{"error", "Max length reached"}}; - send_response(error_response); - this->auto_chat_engine->clear_context(); - return; - } + auto_chat_engine->generate(meta_info, length_limit, ostream, + [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -832,9 +863,11 @@ void RestHandler::handle_chat(const json& request, //std::string response_text = auto_chat_engine->generate_with_prompt(meta_info, uniformed_input, length_limit, std::cout); std::string response_text; try { - response_text = auto_chat_engine->generate_with_prompt(meta_info, uniformed_input, length_limit, nstream); + response_text = auto_chat_engine->generate_with_prompt( + meta_info, uniformed_input, length_limit, nstream, + [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -866,7 +899,7 @@ void RestHandler::handle_chat(const json& request, this->auto_chat_engine->clear_context(); } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -924,7 +957,7 @@ void RestHandler::handle_embeddings(const json& request, send_response(response); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -940,7 +973,7 @@ void RestHandler::handle_models(const json& request, json models = supported_models.get_all_models_ollama(); send_response(models); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -1029,7 +1062,7 @@ void RestHandler::handle_ps(const json& request, // std::cout << "response: " << response.dump(4) << std::endl; send_response(response); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); } } @@ -1186,6 +1219,13 @@ void RestHandler::handle_openai_chat_completion(const json& request, lm_uniform_input_t uniformed_input; uniformed_input.messages = current_messages; uniformed_input.tools = tools; + const std::optional openai_chat_budget = request.contains("max_tokens") + ? std::optional(request.at("max_tokens").get()) + : request.contains("max_completion_tokens") + ? std::optional(request.at("max_completion_tokens").get()) + : std::nullopt; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens(openai_chat_budget); meta_info.load_duration = (uint64_t)time_utils::duration_ns(load_start_time, load_end_time).first; meta_info.max_prefill_len = this->prefill_chunk_len; meta_info.tool_choice = tool_choice_mode; @@ -1193,7 +1233,9 @@ void RestHandler::handle_openai_chat_completion(const json& request, // Create a wrapper callback that passes the pre-formatted SSE string directly cancellation_token->reset(); auto_chat_engine->reset_parser(); - auto openai_stream_callback = [&send_streaming_response](const std::string& data, bool is_final) { + bool stream_started = false; + auto openai_stream_callback = [&send_streaming_response, &stream_started](const std::string& data, bool is_final) { + stream_started = true; json data_json = data; send_streaming_response(data_json, is_final); }; @@ -1225,7 +1267,7 @@ void RestHandler::handle_openai_chat_completion(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); this->prompt_cache.reset(); @@ -1234,8 +1276,18 @@ void RestHandler::handle_openai_chat_completion(const json& request, header_print("FLM", "Start generating..."); try { auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token->cancelled(); }); + } catch (const ModelRequestError& error) { + const json error_response = ModelErrorResponse(error); + if (stream_started) { + send_streaming_response(json("data: " + error_response.dump() + "\n\n"), false); + send_streaming_response(json("data: [DONE]\n\n"), true); + } else { + send_response(error_response); + } + if (error.session_cleared()) this->prompt_cache.reset(); + return; } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); this->prompt_cache.reset(); @@ -1277,7 +1329,7 @@ void RestHandler::handle_openai_chat_completion(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); this->prompt_cache.reset(); @@ -1287,7 +1339,7 @@ void RestHandler::handle_openai_chat_completion(const json& request, try { response_text = auto_chat_engine->generate(meta_info, length_limit, nstream, [&] { return cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); this->prompt_cache.reset(); @@ -1430,18 +1482,25 @@ void RestHandler::handle_openai_completion(const json& request, chat_meta_info_t meta_info; meta_info.max_prefill_len = this->prefill_chunk_len; lm_uniform_input_t uniformed_input; + uniformed_input.requested_max_new_tokens = + normalize_requested_max_new_tokens( + request.contains("max_tokens") + ? std::optional(request.at("max_tokens").get()) + : std::nullopt); header_print("FLM", "Start generating..."); if (stream) { // Create a wrapper callback that passes the pre-formatted SSE string directly - auto openai_stream_callback = [&send_streaming_response](const std::string& data, bool is_final) { + bool stream_started = false; + auto openai_stream_callback = [&send_streaming_response, &stream_started](const std::string& data, bool is_final) { + stream_started = true; json data_json = data; send_streaming_response(data_json, is_final); }; streaming_ostream_openai ostream(model, openai_stream_callback); // streaming in completion format uniformed_input.prompt = prompt; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success) { json error_response = { {"error", "Max length reached"} }; send_response(error_response); @@ -1449,15 +1508,24 @@ void RestHandler::handle_openai_completion(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - auto_chat_engine->generate(meta_info, length_limit, ostream); + auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token && cancellation_token->cancelled(); }); + } catch (const ModelRequestError& error) { + const json error_response = ModelErrorResponse(error); + if (stream_started) { + send_streaming_response(json("data: " + error_response.dump() + "\n\n"), false); + send_streaming_response(json("data: [DONE]\n\n"), true); + } else { + send_response(error_response); + } + return; } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; @@ -1472,7 +1540,7 @@ void RestHandler::handle_openai_completion(const json& request, std::ostream ostream(&obuf); uniformed_input.prompt = prompt; try { - bool success = auto_chat_engine->insert(meta_info, uniformed_input); + bool success = auto_chat_engine->insert(meta_info, uniformed_input, [&] { return cancellation_token && cancellation_token->cancelled(); }); if (!success) { json error_response = { {"error", "Max length reached"} }; send_response(error_response); @@ -1480,15 +1548,15 @@ void RestHandler::handle_openai_completion(const json& request, return; } } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; } try { - auto_chat_engine->generate(meta_info, length_limit, ostream); + auto_chat_engine->generate(meta_info, length_limit, ostream, [&] { return cancellation_token && cancellation_token->cancelled(); }); } catch (const std::exception& e) { - json error_response = {{"error", e.what()}}; + json error_response = ExceptionResponse(e); send_response(error_response); this->auto_chat_engine->clear_context(); return; diff --git a/src/server/rest_handler.hpp b/src/server/rest_handler.hpp index 69256de3a..83730a98e 100644 --- a/src/server/rest_handler.hpp +++ b/src/server/rest_handler.hpp @@ -1,139 +1,147 @@ -/*! - * Copyright (c) 2026 Advanced Micro Devices, Inc. - * \file rest_handler.hpp - * \brief RestHandler class and related declarations - * \author FastFlowLM Team - * \date 2025-06-24 - * \version 0.9.24 - */ -#pragma once - -#include "AutoModel/all_models.hpp" -#ifndef FASTFLOWLM_LINUX_LIMITED_MODELS -#include "whisper/modeling_whisper.hpp" -#include "AutoEmbeddingModel/all_embedding_model.hpp" -#endif -#include "model_list.hpp" -#include "program_args.hpp" - - -#include "model_downloader.hpp" -#include -#include -#include -#include -#include "prompt_cache.hpp" - -using json = nlohmann::ordered_json; - -// Forward declaration -struct CancellationToken; - -///@brief Stream callback type for sending streaming responses -using StreamResponseCallback = std::function; // data, is_final - -class RestHandler { -public: - RestHandler(model_list& models, ModelDownloader& downloader, program_args_t& args); - ~RestHandler(); - - void handle_show(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response); - - void handle_generate(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response, - std::shared_ptr cancellation_token = nullptr); - - void handle_chat(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response, - std::shared_ptr cancellation_token = nullptr); - - - void handle_embeddings(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response); - - - void handle_models(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response); - - void handle_models_openai(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response); - - void handle_ps(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response); - - void handle_version(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response); - - // Placeholder handlers for unimplemented endpoints - void handle_pull(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response); - - void handle_push(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response); - - void handle_delete(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response); - - void handle_copy(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response); - - void handle_create(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response); - - void handle_openai_chat_completion(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response, - std::shared_ptr cancellation_token = nullptr); - void handle_openai_audio_transcriptions(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response, - std::shared_ptr cancellation_token = nullptr); - void handle_openai_completion(const json& request, - std::function send_response, - StreamResponseCallback send_streaming_response, - std::shared_ptr cancellation_token = nullptr); - -private: - bool ensure_model_loaded(const std::string& model_tag); - void ensure_asr_model_loaded(const std::string& model_tag); - void ensure_embed_model_loaded(const std::string& model_tag); - void configure_chat_engine_parameters(const json& options, const json& request); - json build_nstream_response(std::string response_text, chat_meta_info_t& meta_info); - - - std::unique_ptr auto_chat_engine; -#ifndef FASTFLOWLM_LINUX_LIMITED_MODELS - std::unique_ptr whisper_engine; - std::unique_ptr auto_embedding_engine; -#endif - flm_rt::device npu_device_inst; - model_list& supported_models; - ModelDownloader& downloader; - std::string current_model_tag; - std::string default_model_tag; - bool modelscope; - bool asr; - bool embed; - int prefill_chunk_len; - int generate_context_id; - int chat_context_id; - int ctx_length; - int img_pre_resize; - std::string last_question; - bool preemption; - PromptCache prompt_cache; +/*! + * Copyright (c) 2026 Advanced Micro Devices, Inc. + * \file rest_handler.hpp + * \brief RestHandler class and related declarations + * \author FastFlowLM Team + * \date 2025-06-24 + * \version 0.9.24 + */ +#pragma once + +#include "AutoModel/all_models.hpp" +#ifndef FASTFLOWLM_LINUX_LIMITED_MODELS +#include "whisper/modeling_whisper.hpp" +#include "AutoEmbeddingModel/all_embedding_model.hpp" +#endif +#include "model_list.hpp" +#include "program_args.hpp" + + +#include "model_downloader.hpp" +#include +#include +#include +#include +#include "prompt_cache.hpp" + +using json = nlohmann::ordered_json; + +// Forward declaration +struct CancellationToken; + +///@brief Stream callback type for sending streaming responses +using StreamResponseCallback = std::function; // data, is_final + +class RestHandler { +public: + RestHandler(model_list& models, ModelDownloader& downloader, program_args_t& args, + flm_rt::device* npu_device); + ~RestHandler(); + + void handle_show(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response); + + void handle_generate(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response, + std::shared_ptr cancellation_token = nullptr); + + void handle_chat(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response, + std::shared_ptr cancellation_token = nullptr); + + + void handle_embeddings(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response); + + + void handle_models(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response); + + void handle_models_openai(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response); + + void handle_ps(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response); + + void handle_version(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response); + + // Placeholder handlers for unimplemented endpoints + void handle_pull(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response); + + void handle_push(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response); + + void handle_delete(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response); + + void handle_copy(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response); + + void handle_create(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response); + + void handle_openai_chat_completion(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response, + std::shared_ptr cancellation_token = nullptr); + void handle_openai_audio_transcriptions(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response, + std::shared_ptr cancellation_token = nullptr); + void handle_openai_completion(const json& request, + std::function send_response, + StreamResponseCallback send_streaming_response, + std::shared_ptr cancellation_token = nullptr); + +private: + bool ensure_model_loaded(const std::string& model_tag, + const std::string& request_backend = ""); + void ensure_asr_model_loaded(const std::string& model_tag); + void ensure_embed_model_loaded(const std::string& model_tag); + void configure_chat_engine_parameters(const json& options, const json& request); + json build_nstream_response(std::string response_text, chat_meta_info_t& meta_info); + + + std::unique_ptr auto_chat_engine; +#ifndef FASTFLOWLM_LINUX_LIMITED_MODELS + std::unique_ptr whisper_engine; + std::unique_ptr auto_embedding_engine; +#endif + // Owned by main(); on a rai build this is corelib's device, which must + // not be duplicated. + flm_rt::device* npu_device_inst; + model_list& supported_models; + ModelDownloader& downloader; + std::string current_model_tag; + std::string default_model_tag; + bool modelscope; + bool asr; + bool embed; + int prefill_chunk_len; + int generate_context_id; + int chat_context_id; + int ctx_length; + int img_pre_resize; + std::string last_question; + bool preemption; + /// \brief the --backend value, empty when the flag was not given + std::string backend; + /// \brief the backend the currently loaded model was asked for + std::string current_backend; + PromptCache prompt_cache; }; \ No newline at end of file diff --git a/src/server/server.cpp b/src/server/server.cpp index bc6122115..471010b6f 100644 --- a/src/server/server.cpp +++ b/src/server/server.cpp @@ -156,19 +156,6 @@ int NPUAccessManager::get_active_npu_requests() { return g_npu_active_requests.load(); } -// Helper function to check if an endpoint requires NPU access -bool requires_npu_access(const std::string& method, const std::string& path) { - // NPU-intensive endpoints that should be restricted to one user at a time - if (method == "POST") { - return path == "/api/generate" || - path == "/api/chat" || - path == "/v1/chat/completions" || - path == "/v1/audio/transcriptions" || - path == "/v1/embeddings"; - } - return false; -} - ///@brief HttpSession class implementation ///@param socket the socket ///@param server the server @@ -591,37 +578,15 @@ void WebServer::do_accept() { ///@brief process_next_npu_request Handles one queued NPU task at a time void WebServer::process_next_npu_request() { - { - std::lock_guard lock(npu_queue_mutex_); - if (npu_request_queue_.empty()) { - NPUAccessManager::release_npu_access(); - return; // Queue is empty, NPU is free - } - } - - // NPU cooldown before running the next queued task. - constexpr auto npu_cooldown = std::chrono::milliseconds(333); - std::this_thread::sleep_for(npu_cooldown); - - std::function task; - size_t remaining = 0; - { - std::lock_guard lock(npu_queue_mutex_); - if (npu_request_queue_.empty()) { - NPUAccessManager::release_npu_access(); - return; - } - - task = npu_request_queue_.front(); - npu_request_queue_.pop(); - remaining = npu_request_queue_.size(); - } - - header_print("🟡 ", "Dequeuing NPU request (" + std::to_string(remaining) + " remaining)..."); - - // Post the task to be executed by the io_context - net::post(ioc, task); - + npu_request_coordinator_.complete_current( + [this](NPURequestCoordinator::Task task) { + const auto remaining = npu_request_coordinator_.size(); + header_print("🟡 ", "Dequeuing NPU request (" + + std::to_string(remaining) + " remaining)..."); + net::post(ioc, std::move(task)); + }, + [] { NPUAccessManager::release_npu_access(); }, + std::chrono::milliseconds(333)); } ///@brief handle request @@ -684,6 +649,9 @@ bool WebServer::handle_request(http::request& req, auto process_task = [this, it, req_ptr, res_ptr, session, needs_npu, key, is_json](bool is_deferred) { auto& req_ref = *req_ptr; auto& res_ref = *res_ptr; + NPURequestCompletionGuard completion([this, needs_npu] { + if (needs_npu) process_next_npu_request(); + }); // Parse JSON request body json request_json; @@ -704,10 +672,6 @@ bool WebServer::handle_request(http::request& req, // Only write from callback when deferred if (is_deferred && session) session->write_response_from_callback(); - - if (needs_npu) { - this->process_next_npu_request(); - } return; } @@ -737,10 +701,9 @@ bool WebServer::handle_request(http::request& req, if (code == 400) { status = http::status::bad_request; + } else if (code == 500) { + status = http::status::internal_server_error; } - //else if () { - - //} } response_ref.result(status); @@ -750,10 +713,6 @@ bool WebServer::handle_request(http::request& req, cancellation_token->complete(); unregister_active_request(request_id); - if (needs_npu) { - this->process_next_npu_request(); - } - if (is_deferred && session) { session->write_response_from_callback(); } @@ -769,10 +728,6 @@ bool WebServer::handle_request(http::request& req, } if (is_final) { unregister_active_request(request_id); - - if (needs_npu) { - this->process_next_npu_request(); - } } }; @@ -788,10 +743,6 @@ bool WebServer::handle_request(http::request& req, res_ref.set(http::field::content_type, "application/json"); res_ref.prepare_payload(); - if (needs_npu) { - this->process_next_npu_request(); - } - if (is_deferred && session) { session->write_response_from_callback(); } @@ -805,10 +756,6 @@ bool WebServer::handle_request(http::request& req, res_ref.set(http::field::content_type, "application/json"); res_ref.prepare_payload(); - if (needs_npu) { - this->process_next_npu_request(); - } - if (is_deferred && session) { session->write_response_from_callback(); } @@ -831,28 +778,25 @@ bool WebServer::handle_request(http::request& req, return false; } - //const int NPU_QUEUE_LIMIT = 10; - std::lock_guard lock(npu_queue_mutex_); - - if (npu_request_queue_.size() >= max_npu_queue_) { + if (!npu_request_coordinator_.try_enqueue([this, process_task]() { + process_task(true); + })) { res.result(http::status::service_unavailable); res.body() = json{ - {"error", "NPU is in use and request queue is full (limit: " + std::to_string(max_npu_queue_) + "). Please try again later."} + {"error", "NPU is in use and request queue is full (limit: " + + std::to_string(npu_request_coordinator_.capacity()) + + "). Please try again later."} }.dump(); res.set(http::field::content_type, "application/json"); res.prepare_payload(); header_print("🚫 ", "NPU busy and queue full, request denied: " + key); return false; } - else { - // Create a new lambda to bind process_task(true) - npu_request_queue_.push([this, process_task]() { - process_task(true); - }); - header_print("🕒 ", "NPU busy, request queued (" + std::to_string(npu_request_queue_.size()) + "/" + std::to_string(max_npu_queue_) + "): " + key); - return true; - } + header_print("🕒 ", "NPU busy, request queued (" + + std::to_string(npu_request_coordinator_.size()) + "/" + + std::to_string(npu_request_coordinator_.capacity()) + "): " + key); + return true; } ///@brief create lm server @@ -861,9 +805,10 @@ bool WebServer::handle_request(http::request& req, ///@param default_tag the default tag ///@param port the port ///@return the server -std::unique_ptr create_lm_server(model_list& models, ModelDownloader& downloader, program_args_t& args) { +std::unique_ptr create_lm_server(model_list& models, ModelDownloader& downloader, program_args_t& args, + flm_rt::device* npu_device) { auto server = std::make_unique(args.host, args.port, args.cors); - auto rest_handler = std::make_shared(models, downloader, args); + auto rest_handler = std::make_shared(models, downloader, args, npu_device); // Register Ollama-compatible routes server->register_handler("POST", "/api/show", diff --git a/src/server/server.hpp b/src/server/server.hpp index 910c58842..8d3d39d4d 100644 --- a/src/server/server.hpp +++ b/src/server/server.hpp @@ -47,8 +47,88 @@ extern std::mutex g_npu_access_mutex; extern std::atomic g_npu_in_use; extern std::atomic g_npu_active_requests; -// Helper function to check if an endpoint requires NPU access -bool requires_npu_access(const std::string& method, const std::string& path); +// Helper function to check if an endpoint requires serialized accelerator access. +inline bool requires_npu_access(const std::string& method, const std::string& path) { + if (method != "POST") return false; + return path == "/api/generate" || path == "/api/chat" || + path == "/v1/chat/completions" || path == "/v1/completions" || + path == "/v1/audio/transcriptions" || path == "/v1/embeddings"; +} + +class NPURequestCoordinator final { +public: + using Task = std::function; + using Scheduler = std::function; + + explicit NPURequestCoordinator(std::size_t capacity = 10) + : capacity_(capacity) {} + void set_capacity(std::size_t capacity) { + std::lock_guard lock(mutex_); + capacity_ = capacity; + } + bool try_enqueue(Task task) { + std::lock_guard lock(mutex_); + if (tasks_.size() >= capacity_) return false; + tasks_.push(std::move(task)); + return true; + } + Task take_next() { + std::lock_guard lock(mutex_); + if (tasks_.empty()) return {}; + auto task = std::move(tasks_.front()); + tasks_.pop(); + return task; + } + void complete_current(const Scheduler& schedule, + const std::function& release, + std::chrono::milliseconds cooldown) { + auto task = take_next(); + if (!task) { + release(); + return; + } + if (cooldown.count() > 0) std::this_thread::sleep_for(cooldown); + schedule(std::move(task)); + } + bool empty() const { + std::lock_guard lock(mutex_); + return tasks_.empty(); + } + std::size_t size() const { + std::lock_guard lock(mutex_); + return tasks_.size(); + } + std::size_t capacity() const { + std::lock_guard lock(mutex_); + return capacity_; + } +private: + mutable std::mutex mutex_; + std::queue tasks_; + std::size_t capacity_; +}; + +class NPURequestCompletionGuard final { +public: + explicit NPURequestCompletionGuard(std::function completion) + : completion_(std::move(completion)) {} + NPURequestCompletionGuard(const NPURequestCompletionGuard&) = delete; + NPURequestCompletionGuard& operator=(const NPURequestCompletionGuard&) = delete; + NPURequestCompletionGuard(NPURequestCompletionGuard&& other) noexcept + : completion_(std::move(other.completion_)), active_(other.active_) { + other.active_ = false; + } + NPURequestCompletionGuard& operator=(NPURequestCompletionGuard&&) = delete; + ~NPURequestCompletionGuard() { complete(); } + void complete() noexcept { + if (!active_) return; + active_ = false; + try { if (completion_) completion_(); } catch (...) {} + } +private: + std::function completion_; + bool active_ = true; +}; ///@brief get current time string, format: hh:mm:ss mm:dd:yyyy ///@return the current time string @@ -116,7 +196,7 @@ class WebServer { void set_max_connections(size_t max_conns) { max_connections_ = max_conns; } void set_request_timeout(std::chrono::seconds timeout) { request_timeout_ = timeout; } void set_io_threads(size_t num_threads) { io_thread_count_ = num_threads; } - void set_npu_queue_length(size_t q_len) { max_npu_queue_ = q_len; } + void set_npu_queue_length(size_t q_len) { npu_request_coordinator_.set_capacity(q_len); } // Maximum accepted HTTP request body size (in bytes) void set_max_body_size_bytes(std::size_t bytes) { max_body_size_bytes_ = bytes; } std::size_t get_max_body_size_bytes() const { return max_body_size_bytes_; } @@ -161,7 +241,6 @@ class WebServer { std::chrono::seconds request_timeout_ = std::chrono::seconds(600); // 5 minutes size_t io_thread_count_ = 5; std::size_t max_body_size_bytes_ = 256ull * 1024 * 1024; // 256 MB default - size_t max_npu_queue_ = 10; // Request tracking mutable std::mutex active_requests_mutex_; @@ -170,8 +249,7 @@ class WebServer { // Connection tracking std::atomic active_connections_{0}; std::vector io_threads_; - std::queue> npu_request_queue_; - std::mutex npu_queue_mutex_; + NPURequestCoordinator npu_request_coordinator_; // Friend declaration for HttpSession to access private members friend class HttpSession; }; @@ -218,5 +296,6 @@ class model_list; ///@param default_tag the default tag ///@param port the port to listen on, default is 52625, same with the ollama server ///@return the server -std::unique_ptr create_lm_server(model_list& models, ModelDownloader& downloader, program_args_t& args); +std::unique_ptr create_lm_server(model_list& models, ModelDownloader& downloader, program_args_t& args, + flm_rt::device* npu_device); diff --git a/src/src/benchmarking.hpp b/src/src/benchmarking.hpp index 83858a740..bca064c44 100644 --- a/src/src/benchmarking.hpp +++ b/src/src/benchmarking.hpp @@ -246,7 +246,7 @@ void print_result(const BenchmarkResults_t& results) { std::cout << "\n"; } -BenchmarkResults_t run_benchmarks(std::string model_tag, std::string bench_config_file, model_list& availble_models, int iterations){ +BenchmarkResults_t run_benchmarks(std::string model_tag, std::string bench_config_file, model_list& availble_models, int iterations, flm_rt::device* npu_device, const std::string& backend = ""){ BenchmarkResults_t results; json bench_config; // this is used for our benchmarking, not for public use. @@ -267,14 +267,13 @@ BenchmarkResults_t run_benchmarks(std::string model_tag, std::string bench_confi input_file.close(); } - flm_rt::device npu_device_inst = flm_rt::device(0); std::unique_ptr auto_chat_engine; if (!availble_models.is_model_supported(model_tag)) { header_print_r("ERROR", "Model not found: " << model_tag << "; Please check with `flm list` and try again."); return results; } auto [new_tag, model_info] = availble_models.get_model_info(model_tag); - std::pair> auto_model = get_auto_model(new_tag, availble_models, &npu_device_inst); + std::pair> auto_model = get_auto_model(new_tag, availble_models, npu_device); auto_chat_engine = std::move(auto_model.second); bool single_turn = model_info.contains("label") && @@ -285,7 +284,7 @@ BenchmarkResults_t run_benchmarks(std::string model_tag, std::string bench_confi int max_len = bench_config["max_length"]; if (!single_turn && max_len < 8192) max_len = 8192; - auto_chat_engine->load_model(availble_models.get_model_path(model_tag), model_info, max_len, false); + auto_chat_engine->load_model(availble_models.get_model_path(model_tag), model_info, max_len, false, backend); std::string input_text = bench_config["input_text"]; auto [num_tokens, benchmark_text] = auto_chat_engine->prepare_benchmark(input_text); diff --git a/src/src/main.cpp b/src/src/main.cpp index 05e21f0f7..b66d2a96a 100644 --- a/src/src/main.cpp +++ b/src/src/main.cpp @@ -11,6 +11,7 @@ #include "model_downloader.hpp" #include "update.hpp" #include "utils/utils.hpp" +#include "utils/npu_platform.hpp" #include "program_args.hpp" #include "minja/chat-template.hpp" #include @@ -34,6 +35,10 @@ #include "utils/vm_args.hpp" #include #include "benchmarking.hpp" +#ifdef FLM_ENABLE_RAI +#include "rai/corelib_runtime.hpp" +#include "rai/corelib_device.hpp" +#endif #ifndef _WIN32 #include @@ -162,8 +167,9 @@ void signal_handler(int signal) { ///@param models the model list ///@param default_tag the default tag ///@param port the port to listen on, default is 52625, same with the ollama server +///@param npu_device the NPU device owned by main, shared with the handler ///@return the server -std::unique_ptr create_lm_server(model_list& models, ModelDownloader& downloader, program_args_t& args); +std::unique_ptr create_lm_server(model_list& models, ModelDownloader& downloader, program_args_t& args, flm_rt::device* npu_device); #ifdef _WIN32 std::string get_driver_version(const std::string& device_name) { @@ -222,6 +228,9 @@ static bool sanity_check_npu_stack(bool quiet, bool json_output = false) { {"ready", true} }; validation_json["platform"] = "linux"; + // The same build-time generation the catalog was filtered for in main(). + validation_json["npu_platform"] = + std::string(utils::platform_id(utils::build_npu_platform())); // Check kernel version struct utsname u_name; if (uname(&u_name) != 0) { @@ -384,6 +393,10 @@ static bool sanity_check_npu_stack(bool quiet, bool json_output = false) { } validation_json["memlock_ok"] = memlock_ok; + if (print_human) { + header_print_g("Linux", "NPU platform: " << validation_json["npu_platform"].get()); + } + bool overall_ok = amd_device_found && kernel_ok && all_fw_ok && enough_cols && memlock_ok; validation_json["ready"] = overall_ok; if (json_output) { @@ -397,6 +410,8 @@ static bool sanity_check_npu_stack(bool quiet, bool json_output = false) { {"platform", "windows"}, {"amd_device_found", true}, {"npu_driver_ok", true}, + {"npu_platform", + std::string(utils::platform_id(utils::build_npu_platform()))}, {"ready", true} }; std::string npu_arch = identify_npu_arch(); @@ -433,6 +448,7 @@ static bool sanity_check_npu_stack(bool quiet, bool json_output = false) { if (print_human) { header_print_g("Windows", "NPU: " << npu_arch); header_print_g("Windows", "NPU dirver version: " << drv); + header_print_g("Windows", "NPU platform: " << validation_json["npu_platform"].get()); } if (json_output) { @@ -443,6 +459,44 @@ static bool sanity_check_npu_stack(bool quiet, bool json_output = false) { } +#ifdef FLM_ENABLE_RAI +/// \brief brings corelib up for the process and tears it down on every exit path +/// \note main() has early `return 1`s and a catch-all, so the teardown has to be +/// a destructor rather than a trailing call. A failed shutdown must not +/// mask whatever the program was already reporting, hence the swallow. +struct RaiProcessGuard { + ~RaiProcessGuard() { + try { + flm::corelib::CorelibRuntime::ShutdownProcess(); + } catch (const std::exception& e) { + std::cerr << "Warning: corelib shutdown failed: " << e.what() << std::endl; + } catch (...) { + std::cerr << "Warning: corelib shutdown failed" << std::endl; + } + } +}; +#else +///@brief open the NPU device that every engine in this process shares +///@return the shared device, or nullptr when no NPU could be opened +///@note Function-local static, so the lifetime is tied to the process exactly as +/// the per-Runner devices used to be. A machine with no NPU must still be +/// able to run `flm list`/`pull`/`version`, so failure is a null pointer, +/// not an error. +static flm_rt::device* open_npu_device() { + try { + static flm_rt::device npu_device = flm_rt::device(0); + return &npu_device; + } catch (const std::exception& e) { + DO_VERBOSE(1, { + header_print("FLM", "No NPU device available: " << e.what()); + }); + return nullptr; + } catch (...) { + return nullptr; + } +} +#endif + ///@brief main function ///@param argc the number of arguments ///@param argv the arguments @@ -499,8 +553,60 @@ int main(int argc, char* argv[]) { // Get the models directory from environment variable or default std::string models_dir = utils::get_models_directory(); - - model_list availble_models(config_path, models_dir); + // One NPU runtime for the whole process, brought up before the catalog so + // the catalog can be filtered to what this NPU generation can actually run. + // Which runtime that is follows from the build, not from a decision here: + // the flm backend opens its own device, rai gets one from corelib. +#ifdef FLM_ENABLE_RAI + // Declared before anything that can return so the destructor covers the + // early exits and the catch-all below. + RaiProcessGuard rai_guard; + try { + flm::corelib::CorelibRuntime::GetOrCreate(std::filesystem::path(exe_dir)); + } catch (const std::exception& e) { + DO_VERBOSE(1, { header_print("FLM", "corelib unavailable: " << e.what()); }); + } + // TODO: FIXME - corelib's device is not a drop-in for the flm device the + // AutoModel engines expect; the two ownership models conflict, so there is + // no supported way to hand corelib's device out here yet. Until that is + // resolved a rai build reports no flm device, which is harmless because + // the rai backend does not drive the NPU through one. + // flm_rt::device* npu_device = + // const_cast(&ryzenai::corelib::GetDevice()); + flm_rt::device* npu_device = nullptr; +#else + flm_rt::device* npu_device = open_npu_device(); +#endif + + // Which generation this binary is for is decided by FLM_ENABLE_RAI at + // build time: the two generations share no engine, so a build has one of + // them and there is nothing to detect. + constexpr utils::npu_platform platform = utils::build_npu_platform(); + + model_list availble_models(config_path, models_dir, + std::string(utils::platform_id(platform))); + + const bool print_status = !parsed_args.json_output && !parsed_args.sub_process_mode; + const bool needs_npu = + parsed_args.command == "run" || parsed_args.command == "serve" || + parsed_args.command == "bench" || parsed_args.command == "validate"; + if (print_status && needs_npu) { + header_print("FLM", "NPU platform: " << utils::platform_id(platform)); + } + + // The rai build of phi4-mini-it installs under its own directory name, so + // on any other platform that directory is no longer reachable by a tag and + // `flm remove` cannot clean it up. Point it out; never delete it. + if (print_status && platform != utils::npu_platform::aie_next) { + const std::filesystem::path stale_dir = + std::filesystem::path(availble_models.get_model_root_path()) / "phi4-mini-it-rai"; + std::error_code stale_ec; + if (std::filesystem::exists(stale_dir, stale_ec)) { + header_print("FLM", "Note: " << stale_dir.string() + << " is a rai-only model and is unused on this NPU; " + "delete it manually to reclaim the space."); + } + } // Extract parsed values bool got_power_mode = (parsed_args.power_mode != "performance"); // Check if user explicitly set power mode @@ -602,11 +708,11 @@ int main(int argc, char* argv[]) { } if (parsed_args.command == "bench") { - benchmarking::BenchmarkResults_t results = benchmarking::run_benchmarks(parsed_args.model_tag, parsed_args.input_file_name, availble_models, parsed_args.iterations); + benchmarking::BenchmarkResults_t results = benchmarking::run_benchmarks(parsed_args.model_tag, parsed_args.input_file_name, availble_models, parsed_args.iterations, npu_device, parsed_args.backend); } else if (parsed_args.command == "run") { check_and_notify_new_version(); - Runner runner(availble_models, downloader, parsed_args); + Runner runner(availble_models, downloader, parsed_args, npu_device); runner.run(); } else if (parsed_args.command == "serve") { @@ -619,7 +725,7 @@ int main(int argc, char* argv[]) { } else { header_print("FLM", "Using user-specified port: " << port); } - auto server = create_lm_server(availble_models, downloader, parsed_args); + auto server = create_lm_server(availble_models, downloader, parsed_args, npu_device); server->set_max_connections(parsed_args.max_socket_connections); // Allow up to 10 concurrent connections server->set_io_threads(10); // Allow up to 5 io threads server->set_npu_queue_length(parsed_args.max_npu_queue); // Allow up to 10 concurrent queue @@ -705,7 +811,7 @@ int main(int argc, char* argv[]) { std::cerr << "Use --help for usage information" << std::endl; return 1; } - // Return 0 if the command is valid + // Return 0 if the command is valid; corelib_guard shuts corelib down. return 0; } catch (const std::exception& e) { // If an error occurs, this will be used to show the error diff --git a/src/test/model_backend/CMakeLists.txt b/src/test/model_backend/CMakeLists.txt new file mode 100644 index 000000000..45689b977 --- /dev/null +++ b/src/test/model_backend/CMakeLists.txt @@ -0,0 +1,53 @@ +# Registry-only tests: no NPU, no prebuilt engine library, so this builds and +# runs on Linux CI as well as Windows. The XRT headers are still needed because +# model_backend.hpp reaches causal_lm.hpp through the device runtime. +cmake_minimum_required(VERSION 3.22) +project(model_backend_tests LANGUAGES CXX) + +set(CMAKE_CXX_STANDARD 20) +set(CMAKE_CXX_STANDARD_REQUIRED ON) + +set(FLM_SOURCE_DIR "${CMAKE_CURRENT_LIST_DIR}/../..") + +if(WIN32) + set(XRT_INCLUDE_DIR "C:/dev/XRT/src/runtime_src/core/include" + CACHE PATH "Where XRT headers live") +else() + find_package(PkgConfig) + if(PkgConfig_FOUND) + pkg_check_modules(XRT xrt) + endif() + if(XRT_FOUND) + set(XRT_INCLUDE_DIR "${XRT_INCLUDE_DIRS}") + else() + set(XRT_INCLUDE_DIR /opt/xilinx/xrt/include CACHE PATH "Where XRT headers live") + endif() +endif() + +# xrt/detail/any.h includes boost/any.hpp. On Linux Boost is on the default +# search path; on Windows it comes from the conda environment, so point at it. +find_path(BOOST_INCLUDE_DIR NAMES boost/any.hpp + HINTS "$ENV{CONDA_PREFIX}/Library/include" + "$ENV{USERPROFILE}/anaconda3/Library/include") + +add_executable(test_model_backend + test_model_backend.cpp + "${FLM_SOURCE_DIR}/common/AutoModel/model_backend.cpp") +target_include_directories(test_model_backend PRIVATE + "${FLM_SOURCE_DIR}/include" + "${XRT_INCLUDE_DIR}") +if(BOOST_INCLUDE_DIR) + target_include_directories(test_model_backend PRIVATE "${BOOST_INCLUDE_DIR}") +endif() + +if(WIN32) + target_link_directories(test_model_backend PRIVATE "${XRT_INCLUDE_DIR}/../lib") +elseif(XRT_FOUND) + target_link_directories(test_model_backend PRIVATE "${XRT_LIBRARY_DIRS}") +else() + target_link_directories(test_model_backend PRIVATE /opt/xilinx/xrt/lib) +endif() +target_link_libraries(test_model_backend PRIVATE xrt_coreutil) + +enable_testing() +add_test(NAME test_model_backend COMMAND test_model_backend) diff --git a/src/test/model_backend/test_model_backend.cpp b/src/test/model_backend/test_model_backend.cpp new file mode 100644 index 000000000..5ea890aff --- /dev/null +++ b/src/test/model_backend/test_model_backend.cpp @@ -0,0 +1,261 @@ +/// \file test_model_backend.cpp +/// \brief The backend registry and the rules for picking a backend +/// \note Deliberately free of NPU hardware and of any prebuilt engine library: +/// every backend here is a stub, so this builds and runs on Linux CI +/// where the phi4_rai suite cannot. register_builtin_backends +/// is stubbed out below for the same reason. +/// \note A backend id names a kernel provider: "flm", "rai", and one day +/// whatever else grows an engine. It is not a platform id -- the +/// resolution rules below are about which provider a build links, +/// not about which silicon it runs on. +#include "AutoModel/model_backend.hpp" +#include "../phi4_rai/test_support.hpp" + +#include +#include +#include + +using flm::backend::BackendContext; +using flm::backend::BackendRegistry; +using flm::backend::BackendTraits; +using flm::backend::kFlmBackendId; +using flm::backend::kRaiBackendId; +using flm::backend::ModelBackend; +using flm::backend::resolve_backend_id; + +/// \brief the builtin set, emptied +/// \note The real one lives in builtin_backends.cpp and pulls in every engine +/// header, and with them the prebuilt libraries. These tests populate the +/// registry themselves, so an empty set is both enough and honest. +namespace flm::backend { +void register_builtin_backends(BackendRegistry&) {} +} // namespace flm::backend + +namespace { + +/// \brief a backend that owns no engine +/// \note engine() is never called by these tests; nothing here has a causal_lm +/// to hand back, and building one needs an NPU. +class StubBackend final : public ModelBackend { +public: + explicit StubBackend(std::string id) : id_(std::move(id)) {} + causal_lm& engine() override { + throw std::runtime_error("stub backend has no engine"); + } + std::string id() const override { return id_; } + +private: + std::string id_; +}; + +flm::backend::BackendFactory StubFactory(std::string id) { + return [id = std::move(id)](const BackendContext&) { + return std::make_unique(id); + }; +} + +/// \brief a registry that is not the process-wide one +BackendRegistry MakeRegistry() { return BackendRegistry(); } + +/// \brief scoped setenv/unsetenv for FLM_BACKEND +class ScopedBackendEnv { +public: + explicit ScopedBackendEnv(const char* value) { +#if defined(_WIN32) + _putenv_s("FLM_BACKEND", value ? value : ""); +#else + if (value) ::setenv("FLM_BACKEND", value, 1); + else ::unsetenv("FLM_BACKEND"); +#endif + } + ~ScopedBackendEnv() { +#if defined(_WIN32) + _putenv_s("FLM_BACKEND", ""); +#else + ::unsetenv("FLM_BACKEND"); +#endif + } +}; + +void test_register_and_create() { + auto registry = MakeRegistry(); + registry.register_backend("phi4", kFlmBackendId, StubFactory("flm")); + registry.register_backend("phi4", kRaiBackendId, StubFactory("rai"), + BackendTraits{false, false, 4096}); + + TEST_REQUIRE(registry.has("phi4", "flm")); + TEST_REQUIRE(!registry.has("phi4", "bogus")); + TEST_REQUIRE(!registry.has("llama3", "flm")); + + // available() is sorted, which is what makes the error messages stable. + const auto ids = registry.available("phi4"); + TEST_REQUIRE(ids.size() == 2); + TEST_REQUIRE(ids[0] == "flm"); + TEST_REQUIRE(ids[1] == "rai"); + TEST_REQUIRE(registry.available("llama3").empty()); + + BackendContext context; + auto backend = registry.create("phi4", "rai", context); + TEST_REQUIRE(backend != nullptr); + TEST_REQUIRE(backend->id() == "rai"); +} + +void test_traits_are_kept_per_backend() { + auto registry = MakeRegistry(); + registry.register_backend("phi4", kFlmBackendId, StubFactory("flm")); + registry.register_backend("phi4", kRaiBackendId, StubFactory("rai"), + BackendTraits{false, false, 4096}); + + // The defaults describe the FastFlowLM NPU engines, i.e. flm. + const auto flm = registry.traits("phi4", kFlmBackendId); + TEST_REQUIRE(flm.needs_npu_xclbin); + TEST_REQUIRE(flm.supports_preemption); + TEST_REQUIRE(flm.max_context_length == 0); + + const auto rai = registry.traits("phi4", kRaiBackendId); + TEST_REQUIRE(!rai.needs_npu_xclbin); + TEST_REQUIRE(!rai.supports_preemption); + TEST_REQUIRE(rai.max_context_length == 4096); +} + +void test_backend_defaults() { + StubBackend backend(kFlmBackendId); + TEST_REQUIRE(backend.detail().empty()); + TEST_REQUIRE(backend.max_decode_length() == 0); + TEST_REQUIRE(backend.supports_preemption()); + TEST_REQUIRE(backend.forwards_past_eos()); + TEST_REQUIRE(!backend.poisoned()); + TEST_REQUIRE(!backend.forced_eos_ids().has_value()); +} + +void test_duplicate_registration_is_rejected() { + auto registry = MakeRegistry(); + registry.register_backend("phi4", kFlmBackendId, StubFactory("first")); + const std::string message = RequireThrows([&] { + registry.register_backend("phi4", kFlmBackendId, StubFactory("second")); + }); + RequireContains(message, "already registered"); + + BackendContext context; + TEST_REQUIRE(registry.create("phi4", kFlmBackendId, context)->id() == "first"); + + RequireThrows([&] { registry.register_backend("", kFlmBackendId, StubFactory("x")); }); + RequireThrows([&] { registry.register_backend("phi4", "", StubFactory("x")); }); + RequireThrows([&] { registry.register_backend("phi4", "x", nullptr); }); +} + +void test_replace_backend_is_the_test_seam() { + auto registry = MakeRegistry(); + registry.register_backend("phi4", kFlmBackendId, StubFactory("real")); + registry.replace_backend("phi4", kFlmBackendId, StubFactory("stub")); + + BackendContext context; + TEST_REQUIRE(registry.create("phi4", kFlmBackendId, context)->id() == "stub"); + TEST_REQUIRE(registry.available("phi4").size() == 1); + + // It also registers a backend that was not there before. + registry.replace_backend("llama3", kFlmBackendId, StubFactory("fresh")); + TEST_REQUIRE(registry.create("llama3", kFlmBackendId, context)->id() == "fresh"); +} + +void test_unknown_id_names_what_exists() { + auto registry = MakeRegistry(); + registry.register_backend("phi4", kFlmBackendId, StubFactory("flm")); + + BackendContext context; + const std::string message = + RequireThrows([&] { registry.create("phi4", "bogus", context); }); + RequireContains(message, "bogus"); + RequireContains(message, "flm"); + + const std::string empty = + RequireThrows([&] { registry.create("llama3", kFlmBackendId, context); }); + RequireContains(empty, "(none)"); +} + +void test_the_build_default_is_the_default() { + auto& registry = BackendRegistry::instance(); + registry.replace_backend("phi4", kFlmBackendId, StubFactory("flm")); + registry.replace_backend("phi4", kRaiBackendId, StubFactory("rai")); + + std::string source; + + // Nothing overrides it, so the build's own default decides. It is passed + // in rather than detected: which kernel provider a binary links is fixed + // at compile time, and this function has no business asking hardware. + TEST_REQUIRE(resolve_backend_id("phi4", "flm", "", &source) == kFlmBackendId); + TEST_REQUIRE(source == "build default"); + TEST_REQUIRE(resolve_backend_id("phi4", "rai", "", &source) == kRaiBackendId); + TEST_REQUIRE(source == "build default"); +} + +void test_resolution_precedence() { + auto& registry = BackendRegistry::instance(); + registry.replace_backend("phi4", kFlmBackendId, StubFactory("flm")); + registry.replace_backend("phi4", kRaiBackendId, StubFactory("rai")); + + std::string source; + { + // FLM_BACKEND beats the build default. + ScopedBackendEnv env(kRaiBackendId); + TEST_REQUIRE(resolve_backend_id("phi4", "flm", "", &source) == kRaiBackendId); + TEST_REQUIRE(source == "FLM_BACKEND"); + + // --backend beats both. + TEST_REQUIRE(resolve_backend_id("phi4", "flm", kFlmBackendId, &source) == + kFlmBackendId); + TEST_REQUIRE(source == "--backend"); + } + + // An empty FLM_BACKEND is the same as an unset one. + ScopedBackendEnv empty(""); + TEST_REQUIRE(resolve_backend_id("phi4", "rai", "", &source) == kRaiBackendId); + TEST_REQUIRE(source == "build default"); +} + +void test_resolution_rejects_with_a_readable_message() { + auto& registry = BackendRegistry::instance(); + registry.replace_backend("llama3", kFlmBackendId, StubFactory("flm")); + + // A family with no engine for the requested provider: the message has to + // name both what was asked for and what the build does have. + const std::string not_built = RequireThrows( + [&] { resolve_backend_id("llama3", "flm", kRaiBackendId); }); + RequireContains(not_built, "--backend"); + RequireContains(not_built, "not compiled into this build"); + RequireContains(not_built, "flm"); + + // Same for a provider nobody has an engine for yet. + const std::string unknown_provider = + RequireThrows([&] { resolve_backend_id("llama3", "gpu"); }); + RequireContains(unknown_provider, "build default"); + RequireContains(unknown_provider, "gpu"); + + // The env var gets named in the message too, so the user can find it. + ScopedBackendEnv env("bogus"); + const std::string from_env = + RequireThrows([&] { resolve_backend_id("llama3", "flm"); }); + RequireContains(from_env, "FLM_BACKEND"); + + // And a family that has nothing at all still says so rather than crashing. + const std::string no_family = + RequireThrows([&] { resolve_backend_id("nosuchfamily", "flm"); }); + RequireContains(no_family, "(none)"); +} + +} // namespace + +int main() { + RunTest(test_register_and_create, "register and create"); + RunTest(test_traits_are_kept_per_backend, "traits are kept per backend"); + RunTest(test_backend_defaults, "backend policy defaults"); + RunTest(test_duplicate_registration_is_rejected, "duplicate registration is rejected"); + RunTest(test_replace_backend_is_the_test_seam, "replace_backend is the test seam"); + RunTest(test_unknown_id_names_what_exists, "unknown id names what exists"); + RunTest(test_the_build_default_is_the_default, "the build default is the default"); + RunTest(test_resolution_precedence, "resolution precedence"); + RunTest(test_resolution_rejects_with_a_readable_message, + "resolution rejects with a readable message"); + std::cout << "All model backend tests passed\n"; + return 0; +} diff --git a/src/test/model_list_platform/CMakeLists.txt b/src/test/model_list_platform/CMakeLists.txt new file mode 100644 index 000000000..2b9e3b78d --- /dev/null +++ b/src/test/model_list_platform/CMakeLists.txt @@ -0,0 +1,20 @@ +# Catalog-only tests: no NPU, no XRT runtime library, so this builds and runs on +# Linux CI as well as Windows. +cmake_minimum_required(VERSION 3.22) +project(model_list_platform_tests LANGUAGES CXX) + +set(CMAKE_CXX_STANDARD 20) +set(CMAKE_CXX_STANDARD_REQUIRED ON) + +set(FLM_SOURCE_DIR "${CMAKE_CURRENT_LIST_DIR}/../..") + +add_executable(test_model_list_platform + test_model_list_platform.cpp + "${FLM_SOURCE_DIR}/common/npu_platform.cpp") +target_include_directories(test_model_list_platform PRIVATE + "${FLM_SOURCE_DIR}/include") +target_compile_definitions(test_model_list_platform PRIVATE + FLM_TEST_MODEL_LIST_PATH="${FLM_SOURCE_DIR}/model_list.json") + +enable_testing() +add_test(NAME test_model_list_platform COMMAND test_model_list_platform) diff --git a/src/test/model_list_platform/test_model_list_platform.cpp b/src/test/model_list_platform/test_model_list_platform.cpp new file mode 100644 index 000000000..99039e561 --- /dev/null +++ b/src/test/model_list_platform/test_model_list_platform.cpp @@ -0,0 +1,211 @@ +/// \file test_model_list_platform.cpp +/// \brief Platform filtering / override merging in model_list, plus the +/// npu_platform helpers and a sweep of the shipped catalog. +/// \note Deliberately free of NPU hardware: everything here is catalog logic, +/// so it builds and runs on Linux CI where the phi4_rai suite +/// cannot. +#include "model_list.hpp" +#include "utils/npu_platform.hpp" +#include "../phi4_rai/test_support.hpp" + +#include +#include +#include + +namespace { + +constexpr const char* kCatalogPath = FLM_TEST_MODEL_LIST_PATH; +constexpr const char* kPhiTag = "phi4-mini-it:4b"; + +/// \brief build a model_list over the shipped catalog for one platform +model_list open_catalog(const std::string& platform) { + std::string path = kCatalogPath; + std::string exe_dir = "."; + return model_list(path, exe_dir, platform); +} + +nlohmann::json read_catalog() { + std::ifstream stream(kCatalogPath); + TEST_REQUIRE(stream.is_open()); + return nlohmann::json::parse(stream); +} + +void test_stx_entry_is_unchanged() { + auto models = open_catalog("stx"); + const auto [tag, info] = models.get_model_info(kPhiTag); + TEST_REQUIRE(tag == kPhiTag); + TEST_REQUIRE(info.at("name") == "Phi4-mini-Instruct-NPU2"); + TEST_REQUIRE(info.contains("ms_url")); + TEST_REQUIRE(!info.contains("file_sources")); + TEST_REQUIRE(info.at("default_context_length") == 32768); + TEST_REQUIRE(info.at("flm_min_version") == "0.9.25"); + // The stx catalog is the full catalog, and the aie_next-only entry is gone. + TEST_REQUIRE(models.is_model_supported("llama3.2:1b")); + TEST_REQUIRE(!models.is_model_supported("phi4-mini-it-rai:4b")); + TEST_REQUIRE(!models.is_model_supported("phi4-mini-it-rai")); +} + +void test_aie_next_entry_is_merged() { + auto models = open_catalog("aie_next"); + TEST_REQUIRE(models.all_tags.size() == 2); + TEST_REQUIRE(models.is_model_supported("phi4-mini-it")); + TEST_REQUIRE(models.is_model_supported(kPhiTag)); + + const auto [tag, info] = models.get_model_info(kPhiTag); + TEST_REQUIRE(tag == kPhiTag); + // The override wins where it speaks... + TEST_REQUIRE(info.at("name") == "phi4-mini-it-rai"); + TEST_REQUIRE(info.at("default_context_length") == 4096); + TEST_REQUIRE(info.at("flm_min_version") == "1.0.3"); + TEST_REQUIRE(info.at("model_info_key") == "phi4-mini-it-rai:4b"); + TEST_REQUIRE(info.at("files").size() == 4); + TEST_REQUIRE(info.at("file_sources").size() == 3); + // ...a null in the patch deletes the key... + TEST_REQUIRE(!info.contains("ms_url")); + // ...and the rest of details survives the recursive merge. + TEST_REQUIRE(info.at("details").at("family") == "phi4"); + TEST_REQUIRE(info.at("details").at("parameter_size") == "4B"); + // Bookkeeping keys never reach the caller. + TEST_REQUIRE(!info.contains("supported_platforms")); + TEST_REQUIRE(!info.contains("platform_overrides")); + // And the entry names no backend: the hardware it was selected for is the + // backend, so there is nothing left for the catalog to say about it. + TEST_REQUIRE(!info.contains("supported_backends")); + TEST_REQUIRE(!info.at("details").contains("execution_backend")); +} + +void test_pruned_lookups_do_not_throw() { + auto models = open_catalog("aie_next"); + // Both of these used to dereference the pruned llama3.2 family. + const auto [missing_tag, missing_info] = models.get_model_info("bogus:9b"); + TEST_REQUIRE(missing_tag == kPhiTag); + TEST_REQUIRE(missing_info.at("default_context_length") == 4096); + TEST_REQUIRE(models.rectify_model_tag("llama3.2") == "llama3.2"); + TEST_REQUIRE(models.rectify_model_tag("phi4-mini-it") == kPhiTag); +} + +void test_platform_helpers() { + TEST_REQUIRE(utils::parse_platform("stx") == utils::npu_platform::stx); + TEST_REQUIRE(utils::parse_platform("aie_next") == utils::npu_platform::aie_next); + TEST_REQUIRE(!utils::parse_platform("not-a-platform").has_value()); + TEST_REQUIRE(utils::parse_platform(utils::platform_id(utils::npu_platform::aie_next)) == + utils::npu_platform::aie_next); + TEST_REQUIRE(utils::default_npu_platform() == utils::npu_platform::stx); + + // The generation is whatever this binary was built for, nothing else. +#ifdef FLM_ENABLE_RAI + TEST_REQUIRE(utils::build_npu_platform() == utils::npu_platform::aie_next); +#else + TEST_REQUIRE(utils::build_npu_platform() == utils::npu_platform::stx); +#endif +} + +void test_shipped_catalog_is_well_formed() { + const auto catalog = read_catalog(); + TEST_REQUIRE(!catalog.at("models").contains("phi4-mini-it-rai")); + + for (const auto& [family, sizes] : catalog.at("models").items()) { + for (const auto& [size, entry] : sizes.items()) { + const std::string tag = family + ":" + size; + + // Nothing in the catalog names a backend any more. + if (entry.contains("supported_backends")) { + throw std::runtime_error(tag + ": supported_backends is retired"); + } + if (entry.contains("details") && + entry.at("details").contains("execution_backend")) { + throw std::runtime_error(tag + ": execution_backend is retired"); + } + + // An entry is only tagged if it runs somewhere other than stx, so + // the common case is no key at all. A key that says only ["stx"] + // is not wrong, just noise, and this keeps it from creeping back. + const nlohmann::json supported = + entry.value("supported_platforms", nlohmann::json::array()); + if (entry.contains("supported_platforms")) { + if (!supported.is_array() || supported.empty()) { + throw std::runtime_error(tag + ": supported_platforms must be a non-empty array"); + } + bool beyond_stx = false; + for (const auto& value : supported) { + if (!value.is_string() || + !utils::parse_platform(value.get()).has_value()) { + throw std::runtime_error(tag + ": unknown platform in supported_platforms"); + } + if (value.get() != "stx") beyond_stx = true; + } + if (!beyond_stx) { + throw std::runtime_error( + tag + ": supported_platforms says only stx, which is " + "the default -- drop the key"); + } + } + if (!entry.contains("platform_overrides")) continue; + const auto& overrides = entry.at("platform_overrides"); + if (!overrides.is_object()) { + throw std::runtime_error(tag + ": platform_overrides must be an object"); + } + for (const auto& [platform, patch] : overrides.items()) { + bool declared = false; + for (const auto& value : supported) { + if (value.get() == platform) declared = true; + } + if (!declared) { + throw std::runtime_error( + tag + ": platform_overrides has '" + platform + + "', which is not in supported_platforms"); + } + if (!patch.is_object()) { + throw std::runtime_error(tag + ": override for '" + platform + + "' must be an object"); + } + } + } + } +} + +void test_missing_key_means_stx_only() { + // An untagged entry is stx-only. That is the rule the shipped catalog + // leans on -- only aie_next support gets a tag -- so it is worth pinning down + // from both sides: the untagged entry must appear on stx and must not + // leak onto aie_next. + const auto path = std::filesystem::temp_directory_path() / + "flm_legacy_model_list.json"; + nlohmann::json untagged = { + {"model_path", "models"}, + {"models", + {{"legacy", {{"1b", {{"name", "Legacy"}}}}}, + {"both", + {{"1b", + {{"name", "Both"}, + {"supported_platforms", {"stx", "aie_next"}}}}}}}}}; + { + std::ofstream out(path); + out << untagged.dump(2); + } + std::string list_path = path.string(); + std::string exe_dir = "."; + + model_list stx(list_path, exe_dir, "stx"); + TEST_REQUIRE(stx.is_model_supported("legacy:1b")); + TEST_REQUIRE(stx.is_model_supported("both:1b")); + + model_list aie_next(list_path, exe_dir, "aie_next"); + TEST_REQUIRE(!aie_next.is_model_supported("legacy:1b")); + TEST_REQUIRE(aie_next.is_model_supported("both:1b")); + + std::filesystem::remove(path); +} + +} // namespace + +int main() { + RunTest(test_stx_entry_is_unchanged, "stx entry is unchanged"); + RunTest(test_aie_next_entry_is_merged, "aie_next entry is merged"); + RunTest(test_pruned_lookups_do_not_throw, "pruned lookups do not throw"); + RunTest(test_platform_helpers, "platform helpers"); + RunTest(test_shipped_catalog_is_well_formed, "shipped catalog is well formed"); + RunTest(test_missing_key_means_stx_only, "missing key means stx only"); + std::cout << "All model_list platform tests passed\n"; + return 0; +} diff --git a/src/test/phi4_rai/CMakeLists.txt b/src/test/phi4_rai/CMakeLists.txt new file mode 100644 index 000000000..8b62fb819 --- /dev/null +++ b/src/test/phi4_rai/CMakeLists.txt @@ -0,0 +1,264 @@ +cmake_minimum_required(VERSION 3.22) +project(phi4_rai_tests LANGUAGES CXX) + +set(CMAKE_CXX_STANDARD 20) +set(CMAKE_CXX_STANDARD_REQUIRED ON) + +if(NOT WIN32) + message(FATAL_ERROR "Phi-4 corelib tests currently require Windows") +endif() + +find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h REQUIRED) +find_path(BOOST_INCLUDE_DIR NAMES boost/program_options.hpp + HINTS "$ENV{CONDA_PREFIX}/Library/include" + "$ENV{USERPROFILE}/anaconda3/Library/include" REQUIRED) +find_path(XRT_INCLUDE_DIR NAMES xrt/xrt_bo.h + HINTS "$ENV{XRT_INCLUDE_DIR}" + "${CMAKE_CURRENT_LIST_DIR}/../../../../xrt_package/xrt/include" + "C:/dev/XRT/src/runtime_src/core/include" REQUIRED) +find_path(CURL_INCLUDE_DIR NAMES curl/curl.h + HINTS "$ENV{CONDA_PREFIX}/Library/include" + "$ENV{USERPROFILE}/anaconda3/Library/include" REQUIRED) +find_library(CURL_LIBRARY NAMES libcurl curl + HINTS "$ENV{CONDA_PREFIX}/Library/lib" + "$ENV{USERPROFILE}/anaconda3/Library/lib" REQUIRED) +find_package(CURL REQUIRED) +set(FLM_SOURCE_DIR "${CMAKE_CURRENT_LIST_DIR}/../..") +set(CORELIB_SOURCES + "${FLM_SOURCE_DIR}/common/rai/corelib_api.cpp" + "${FLM_SOURCE_DIR}/common/rai/corelib_runtime.cpp") + +# A synthetic mismatched copy proves that the adapter rejects header drift at +# compile time. The caller-provided header remains untouched. +file(READ "${RYZENAI_CORELIB_INCLUDE_DIR}/ryzenai/corelib.h" CORELIB_HEADER_TEXT) +string(REGEX REPLACE + "#define RYZENAI_CORELIB_VERSION_PATCH[ \t]+0" + "#define RYZENAI_CORELIB_VERSION_PATCH 1" + WRONG_CORELIB_HEADER_TEXT "${CORELIB_HEADER_TEXT}") +set(WRONG_CORELIB_INCLUDE_DIR + "${CMAKE_CURRENT_BINARY_DIR}/wrong-corelib-version/include") +file(MAKE_DIRECTORY "${WRONG_CORELIB_INCLUDE_DIR}/ryzenai") +file(WRITE "${WRONG_CORELIB_INCLUDE_DIR}/ryzenai/corelib.h" + "${WRONG_CORELIB_HEADER_TEXT}") +file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/wrong-corelib-version/probe.cpp" + "#define RYZENAI_CORELIB_STATIC 1\n#include \"rai/corelib_api.hpp\"\n") +try_compile(CORELIB_WRONG_VERSION_ACCEPTED + SOURCES "${CMAKE_CURRENT_BINARY_DIR}/wrong-corelib-version/probe.cpp" + CMAKE_FLAGS + "-DCMAKE_CXX_STANDARD=20" + "-DCMAKE_CXX_FLAGS=/I${FLM_SOURCE_DIR}/include /I${WRONG_CORELIB_INCLUDE_DIR}" + OUTPUT_VARIABLE WRONG_CORELIB_COMPILE_OUTPUT) +if(CORELIB_WRONG_VERSION_ACCEPTED) + message(FATAL_ERROR "Corelib adapter accepted a header it is not pinned to") +endif() + +add_executable(test_corelib_api + test_corelib_api.cpp fake_corelib.cpp ${CORELIB_SOURCES}) +target_include_directories(test_corelib_api PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}") +target_compile_definitions(test_corelib_api PRIVATE + RYZENAI_CORELIB_STATIC=1 FLM_CORELIB_TESTING=1) + +add_executable(test_real_corelib test_real_corelib.cpp ${CORELIB_SOURCES}) +target_include_directories(test_real_corelib PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}") +target_compile_definitions(test_real_corelib PRIVATE RYZENAI_CORELIB_STATIC=1) + +add_executable(test_phi4_gguf + test_phi4_gguf.cpp fake_corelib.cpp + "${FLM_SOURCE_DIR}/common/rai/corelib_api.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_gguf.cpp") +target_include_directories(test_phi4_gguf PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}") +target_compile_definitions(test_phi4_gguf PRIVATE RYZENAI_CORELIB_STATIC=1) + +add_executable(test_phi4_host + test_phi4_host.cpp + "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_host.cpp") +target_include_directories(test_phi4_host PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include") + +add_executable(test_phi4_shape_plan + test_phi4_shape_plan.cpp fake_corelib.cpp + "${FLM_SOURCE_DIR}/common/rai/corelib_api.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_shape_plan.cpp") +target_include_directories(test_phi4_shape_plan PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}") +target_compile_definitions(test_phi4_shape_plan PRIVATE RYZENAI_CORELIB_STATIC=1) + +add_executable(test_phi4_engine + test_phi4_engine.cpp fake_corelib.cpp + "${FLM_SOURCE_DIR}/common/rai/corelib_api.cpp" + "${FLM_SOURCE_DIR}/common/rai/corelib_runtime.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_gguf.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_host.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_shape_plan.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_weight_cache.cpp") +target_include_directories(test_phi4_engine PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") +target_compile_definitions(test_phi4_engine PRIVATE + RYZENAI_CORELIB_STATIC=1 USEAVX2=1 DISABLE_ABI_CHECK=1 + _ENABLE_EXTENDED_ALIGNED_STORAGE WIN32_LEAN_AND_MEAN NOMINMAX) +target_compile_options(test_phi4_engine PRIVATE + $<$:/wd4005 /wd4244>) +target_link_directories(test_phi4_engine PRIVATE "${XRT_INCLUDE_DIR}/../lib") +target_link_libraries(test_phi4_engine PRIVATE xrt_coreutil) + +# The backend registry is part of the frontend now; builtin_backends.cpp is not, +# because it names every prebuilt engine library. test_phi4_frontend.cpp supplies +# its own empty register_builtin_backends and registers stubs instead. +set(PHI4_FRONTEND_SOURCES + "${FLM_SOURCE_DIR}/common/AutoModel/automodel.cpp" + "${FLM_SOURCE_DIR}/common/AutoModel/model_backend.cpp" + "${FLM_SOURCE_DIR}/common/AutoModel/modeling_phi4.cpp") + +add_executable(test_phi4_frontend + test_phi4_frontend.cpp fake_corelib.cpp + ${PHI4_FRONTEND_SOURCES} + ${CORELIB_SOURCES} + "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_gguf.cpp") +target_include_directories(test_phi4_frontend PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${FLM_SOURCE_DIR}/server" + "${FLM_SOURCE_DIR}/pull" + "${FLM_SOURCE_DIR}/runner" + "${FLM_SOURCE_DIR}/../third_party/tokenizers-cpp/include" + "${RYZENAI_CORELIB_INCLUDE_DIR}" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") +target_compile_definitions(test_phi4_frontend PRIVATE + FLM_ENABLE_RAI=1 FLM_CORELIB_TESTING=1 + RYZENAI_CORELIB_STATIC=1 DEV_BUILD=1 __WINDOWS__ USEAVX2=1 + DISABLE_ABI_CHECK=1 _ENABLE_EXTENDED_ALIGNED_STORAGE + WIN32_LEAN_AND_MEAN NOMINMAX) +target_compile_options(test_phi4_frontend PRIVATE + $<$:/wd4005 /wd4244>) +target_link_directories(test_phi4_frontend PRIVATE "${XRT_INCLUDE_DIR}/../lib") +target_link_libraries(test_phi4_frontend PRIVATE xrt_coreutil) + +add_executable(test_model_downloader + test_model_downloader.cpp + "${FLM_SOURCE_DIR}/pull/download_model.cpp" + "${FLM_SOURCE_DIR}/pull/model_downloader.cpp" + "${FLM_SOURCE_DIR}/common/utils.cpp") +target_include_directories(test_model_downloader PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${FLM_SOURCE_DIR}/pull" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") +target_compile_definitions(test_model_downloader PRIVATE + FLM_SOURCE_DIR="${FLM_SOURCE_DIR}" + CMAKE_INSTALL_PREFIX="${FLM_SOURCE_DIR}/build" + CMAKE_XCLBIN_PREFIX="${FLM_SOURCE_DIR}/xclbins" + __FLM_VERSION__="1.0.3" + __NPU_VERSION__="0.0.0.0" + DEV_BUILD=1 __WINDOWS__ USEAVX2=1 DISABLE_ABI_CHECK=1 _ENABLE_EXTENDED_ALIGNED_STORAGE + WIN32_LEAN_AND_MEAN NOMINMAX) +target_compile_options(test_model_downloader PRIVATE + $<$:/wd4005 /wd4244>) +target_link_libraries(test_model_downloader PRIVATE CURL::libcurl) + +add_executable(test_phi4_frontend_off + test_phi4_frontend.cpp + ${PHI4_FRONTEND_SOURCES}) +target_include_directories(test_phi4_frontend_off PRIVATE + "${CMAKE_CURRENT_LIST_DIR}" + "${FLM_SOURCE_DIR}/include" + "${FLM_SOURCE_DIR}/server" + "${FLM_SOURCE_DIR}/pull" + "${FLM_SOURCE_DIR}/runner" + "${FLM_SOURCE_DIR}/../third_party/tokenizers-cpp/include" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") +target_compile_definitions(test_phi4_frontend_off PRIVATE + FLM_CORELIB_TESTING=1 DEV_BUILD=1 __WINDOWS__ USEAVX2=1 + DISABLE_ABI_CHECK=1 _ENABLE_EXTENDED_ALIGNED_STORAGE + WIN32_LEAN_AND_MEAN NOMINMAX) +target_compile_options(test_phi4_frontend_off PRIVATE + $<$:/wd4005 /wd4244>) +target_link_directories(test_phi4_frontend_off PRIVATE "${XRT_INCLUDE_DIR}/../lib") +target_link_libraries(test_phi4_frontend_off PRIVATE xrt_coreutil) + +# Compile the actual production frontend translation units in both feature modes. +# Empty declaration-only FFmpeg headers isolate this compile check from an +# unrelated optional SDK that is absent on the standalone test host. +set(FRONTEND_STUB_INCLUDE_DIR "${CMAKE_CURRENT_BINARY_DIR}/frontend-compile-stubs") +foreach(STUB_HEADER IN ITEMS + libavcodec/avcodec.h + libavformat/avformat.h + libswscale/swscale.h + libavutil/imgutils.h + libavutil/frame.h + libavutil/pixfmt.h) + get_filename_component(STUB_PARENT + "${FRONTEND_STUB_INCLUDE_DIR}/${STUB_HEADER}" DIRECTORY) + file(MAKE_DIRECTORY "${STUB_PARENT}") + file(WRITE "${FRONTEND_STUB_INCLUDE_DIR}/${STUB_HEADER}" "#pragma once\n") +endforeach() + +set(FLM_PRODUCTION_FRONTEND_SOURCES + ${PHI4_FRONTEND_SOURCES} + "${FLM_SOURCE_DIR}/common/AutoModel/builtin_backends.cpp" + "${FLM_SOURCE_DIR}/runner/runner.cpp" + "${FLM_SOURCE_DIR}/server/rest_handler.cpp" + "${FLM_SOURCE_DIR}/server/server.cpp") +function(add_frontend_compile_guard TARGET_NAME ENABLE_CORELIB) + add_library(${TARGET_NAME} OBJECT ${FLM_PRODUCTION_FRONTEND_SOURCES}) + target_include_directories(${TARGET_NAME} PRIVATE + "${FLM_SOURCE_DIR}/include" + "${FLM_SOURCE_DIR}/runner" + "${FLM_SOURCE_DIR}/server" + "${FLM_SOURCE_DIR}/pull" + "${FLM_SOURCE_DIR}/../third_party/tokenizers-cpp/include" + "${FRONTEND_STUB_INCLUDE_DIR}" + "${BOOST_INCLUDE_DIR}" + "${XRT_INCLUDE_DIR}") + target_compile_definitions(${TARGET_NAME} PRIVATE + DEV_BUILD=1 __WINDOWS__ USEAVX2=1 DISABLE_ABI_CHECK=1 + _ENABLE_EXTENDED_ALIGNED_STORAGE WIN32_LEAN_AND_MEAN NOMINMAX + __FLM_VERSION__="test" __NPU_VERSION__="0.0.0.0" + CMAKE_INSTALL_PREFIX="${FLM_SOURCE_DIR}/build" + CMAKE_XCLBIN_PREFIX="${FLM_SOURCE_DIR}/xclbins") + target_compile_options(${TARGET_NAME} PRIVATE + $<$:/wd4005 /wd4244>) + if(ENABLE_CORELIB) + target_sources(${TARGET_NAME} PRIVATE + "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_backend.cpp") + target_compile_definitions(${TARGET_NAME} PRIVATE + FLM_ENABLE_RAI=1) + target_include_directories(${TARGET_NAME} PRIVATE + "${RYZENAI_CORELIB_INCLUDE_DIR}") + endif() +endfunction() + +# The OFF target deliberately has no RYZENAI_CORELIB_INCLUDE_DIR. +add_frontend_compile_guard(phi4_frontend_compile_off FALSE) +add_frontend_compile_guard(phi4_frontend_compile_on TRUE) + +include(CTest) +add_test(NAME test_corelib_api COMMAND test_corelib_api) +add_test(NAME test_real_corelib COMMAND test_real_corelib) +add_test(NAME test_phi4_gguf COMMAND test_phi4_gguf) +add_test(NAME test_phi4_host COMMAND test_phi4_host) +add_test(NAME test_phi4_shape_plan COMMAND test_phi4_shape_plan) +add_test(NAME test_phi4_engine COMMAND test_phi4_engine) +add_test(NAME test_phi4_frontend COMMAND test_phi4_frontend) +add_test(NAME test_phi4_frontend_off COMMAND test_phi4_frontend_off) +add_test(NAME test_model_downloader COMMAND test_model_downloader) +set_tests_properties(test_real_corelib PROPERTIES SKIP_RETURN_CODE 77) diff --git a/src/test/phi4_rai/fake_corelib.cpp b/src/test/phi4_rai/fake_corelib.cpp new file mode 100644 index 000000000..251da0118 --- /dev/null +++ b/src/test/phi4_rai/fake_corelib.cpp @@ -0,0 +1,596 @@ +#include "fake_corelib.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +fake_corelib::State state; +std::recursive_mutex state_mutex; +thread_local std::string current_detail; + +struct FakeStorage { + std::size_t byte_size{}; + std::unique_ptr> bytes; +}; + +struct FakeObject { + std::string kind; + /// byte the fake's packed bytes are filled with, so a cached blob can be + /// told apart from a freshly packed one and a round trip can be checked + unsigned char fill{}; + ryzenai_corelib_data_type data_type{ryzenai_corelib_data_type_bf16}; + std::vector shape; + std::size_t byte_size{}; + std::size_t window_offset{}; + std::shared_ptr storage; +}; + +void* NewObject(std::string kind = "generic") { + ++state.live_objects; + auto* object = new FakeObject; + object->kind = std::move(kind); + return object; +} + +/// \brief the size the fake claims a packed weight has +/// \note Only has to be deterministic and descriptor-derived: corelib rejects +/// a cached slice whose length is not exactly what the descriptor packs +/// to, and this is what lets a test exercise that. +std::size_t PackedSize(std::int64_t k, std::int64_t n) { + return static_cast(k) * static_cast(n) / 2 + 64; +} + +/// \brief a byte that identifies which source range a weight was packed from +unsigned char FillFor(const void* source) { + return static_cast( + (reinterpret_cast(source) >> 4) & 0xFF); +} + +ryzenai_corelib_status Status(std::string_view name) { + const auto configured = state.statuses.find(std::string(name)); + return configured == state.statuses.end() ? state.default_status + : configured->second; +} + +std::size_t Elements(const std::vector& shape) { + std::size_t result = 1; + for (const auto dimension : shape) result *= static_cast(dimension); + return result; +} + +std::size_t TypeBytes(ryzenai_corelib_data_type type) { + return RYZENAI_CORELIB_DATA_TYPE_BITS(type) / 8; +} + +std::int64_t PaddedRows(std::string_view helper, std::int64_t rows) { + const auto helpers = state.pad_row_overrides.find(std::string(helper)); + if (helpers != state.pad_row_overrides.end()) { + const auto found = helpers->second.find(rows); + if (found != helpers->second.end()) return found->second; + } + if (rows == 1 || state.pad_multiple <= 0) return rows; + return (rows + state.pad_multiple - 1) / state.pad_multiple * state.pad_multiple; +} + +std::uint16_t Bf16(float value) { + std::uint32_t bits = std::bit_cast(value); + bits += 0x7fffU + ((bits >> 16) & 1U); + return static_cast(bits >> 16); +} + +float FloatFromBf16(std::uint16_t value) { + return std::bit_cast(static_cast(value) << 16); +} + +void EnsureStorage(FakeObject& object) { + if (!object.storage->bytes) + object.storage->bytes = std::make_unique>( + object.storage->byte_size, std::byte{0}); +} + +void ObserveCreateConcurrency() { + const int active = ++state.active_weight_creates; + int maximum = state.maximum_active_weight_creates.load(); + while (active > maximum && + !state.maximum_active_weight_creates.compare_exchange_weak(maximum, active)) {} +} + +/// \brief stand in for the packing a real create spends its time on +/// \param state_lock the fake state mutex, held by the caller +/// \note Every fake entry point holds the state mutex for its whole body, so +/// without releasing it here the creates would serialise no matter how +/// many threads the engine used, and the concurrency could not be +/// observed -- or exercised. Real packing touches only its own mapped +/// range, which is exactly what is modelled by stepping outside. +void SimulatePackingWork(std::unique_lock& state_lock) { + state_lock.unlock(); + std::this_thread::sleep_for(std::chrono::milliseconds(2)); + state_lock.lock(); +} + +#define FLM_DEFINE_FAKE_TAG(member, symbol) \ + struct member##_tag { \ + static constexpr std::string_view name = #symbol; \ + }; +FLM_CORELIB_FUNCTIONS(FLM_DEFINE_FAKE_TAG) +#undef FLM_DEFINE_FAKE_TAG + +template +inline constexpr bool kAlwaysFalse = false; + +template +struct TypedFake; + +template +struct TypedFake { + static Result Invoke(Args... args) { + std::unique_lock state_lock(state_mutex); + ++state.call_counts[std::string(Tag::name)]; + state.call_log.emplace_back(Tag::name); + auto arguments = std::forward_as_tuple(args...); + + if constexpr (std::is_same_v) { + if (std::get<0>(arguments)) *std::get<0>(arguments) = state.version.major; + if (std::get<1>(arguments)) *std::get<1>(arguments) = state.version.minor; + if (std::get<2>(arguments)) *std::get<2>(arguments) = state.version.patch; + return; + } else if constexpr (std::is_same_v) { + current_detail = "detail overwritten by status_to_string"; + return state.status_text.c_str(); + } else if constexpr (std::is_same_v) { + current_detail = state.detail; + return current_detail.c_str(); + } else if constexpr (std::is_same_v) { + return state.selftest_status; + } else if constexpr (std::is_same_v) { + // 0.5.0 answers "is there an NPU" with the device pointer itself, + // NULL when there is none. The fake has no device to hand out, so + // any stable non-null address will do -- nothing dereferences it. + static const int kFakeDevice = 0; + return state.has_device_context + ? static_cast(&kFakeDevice) + : nullptr; + } else if constexpr (std::is_same_v) { + void* object = std::get<0>(arguments); + if (object) { + delete static_cast(object); + --state.live_objects; + ++state.releases; + state.lifetime_events.emplace_back("release"); + } + return; + } else if constexpr (std::is_same_v) { + ++state.cleanup_calls; + state.lifetime_events.emplace_back("cleanup"); + return; + } else if constexpr (std::is_same_v) { + // 0.5.0 signature is (prefill_pdi, token_pdi, out): the PDI pair + // leads, so the out parameter is third. Recorded so a test can + // assert the pair the engine opened its stream with. + const auto status = Status(Tag::name); + state.stream_prefill_pdi = std::get<0>(arguments); + state.stream_token_pdi = std::get<1>(arguments); + auto* out = std::get<2>(arguments); + if (out) *out = status == ryzenai_corelib_status_success ? NewObject("stream") : nullptr; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + const auto type = std::get<0>(arguments); + const auto* shape = std::get<1>(arguments); + const auto shape_len = std::get<2>(arguments); + auto* out = std::get<3>(arguments); + if (out) *out = nullptr; + if (status == ryzenai_corelib_status_success && out && shape) { + auto* object = static_cast(NewObject("tensor")); + object->data_type = type; + object->shape.assign(shape, shape + shape_len); + object->byte_size = Elements(object->shape) * TypeBytes(type); + object->storage = std::make_shared(); + object->storage->byte_size = object->byte_size; + *out = object; + state.tensor_creates.push_back({type, object->shape, object}); + } + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + void* parent = std::get<0>(arguments); + const auto* shape = std::get<1>(arguments); + const auto shape_len = std::get<2>(arguments); + const auto offset = std::get<3>(arguments); + auto* out = std::get<4>(arguments); + if (out) *out = nullptr; + if (status == ryzenai_corelib_status_success && out && shape) { + auto* object = static_cast(NewObject("window")); + if (parent) { + const auto* parent_object = static_cast(parent); + object->data_type = parent_object->data_type; + object->storage = parent_object->storage; + object->window_offset = parent_object->window_offset + offset; + } + object->shape.assign(shape, shape + shape_len); + object->byte_size = Elements(object->shape) * TypeBytes(object->data_type); + *out = object; + state.tensor_windows.push_back({parent, object->shape, offset, object}); + } + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && std::get<0>(arguments) && std::get<1>(arguments)) + *std::get<1>(arguments) = static_cast(std::get<0>(arguments))->byte_size; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && std::get<0>(arguments) && std::get<1>(arguments)) + *std::get<1>(arguments) = static_cast(std::get<0>(arguments))->data_type; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + const auto type = std::get<1>(arguments); + const void* source = std::get<2>(arguments); + const auto count = std::get<3>(arguments); + const auto offset = std::get<4>(arguments); + bool all_zero = true; + if (source) { + const auto* bytes = static_cast(source); + all_zero = std::all_of(bytes, bytes + count * TypeBytes(type), + [](unsigned char value) { return value == 0; }); + } + state.tensor_writes.push_back({std::get<0>(arguments), type, count, offset, all_zero}); + auto* object = static_cast(std::get<0>(arguments)); + if (status == ryzenai_corelib_status_success && object && source) { + const auto target_offset = (object->window_offset + offset) * + TypeBytes(object->data_type); + if (!all_zero || object->storage->bytes) EnsureStorage(*object); + if (object->storage->bytes) { + auto* target = object->storage->bytes->data() + target_offset; + if (object->data_type == type) { + std::memcpy(target, source, count * TypeBytes(type)); + } else if (object->data_type == ryzenai_corelib_data_type_bf16 && + type == ryzenai_corelib_data_type_fp32) { + const auto* values = static_cast(source); + for (std::size_t i = 0; i < count; ++i) { + const auto converted = Bf16(values[i]); + std::memcpy(target + i * sizeof(converted), &converted, + sizeof(converted)); + } + } + } + } + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + auto* object = static_cast(std::get<0>(arguments)); + const auto destination_type = std::get<1>(arguments); + void* destination = std::get<2>(arguments); + const auto count = std::get<3>(arguments); + const auto offset = std::get<4>(arguments); + if (status == ryzenai_corelib_status_success && destination) { + std::memset(destination, 0, count * TypeBytes(destination_type)); + if (object && object->storage && object->storage->bytes) { + const auto source_offset = (object->window_offset + offset) * + TypeBytes(object->data_type); + const auto* source = object->storage->bytes->data() + source_offset; + if (object->data_type == destination_type) { + std::memcpy(destination, source, + count * TypeBytes(destination_type)); + } else if (object->data_type == ryzenai_corelib_data_type_bf16 && + destination_type == ryzenai_corelib_data_type_fp32) { + auto* values = static_cast(destination); + for (std::size_t i = 0; i < count; ++i) { + std::uint16_t encoded; + std::memcpy(&encoded, source + i * sizeof(encoded), + sizeof(encoded)); + values[i] = FloatFromBf16(encoded); + } + } + } + } + return status; + } else if constexpr (std::is_same_v) { + // Every padding helper gained a leading stream in 0.5.0 -- the PDI + // pair it was opened with is what selects the kernel set, so the + // answer is per-stream. Arguments shift by one accordingly. + auto* m = std::get<1>(arguments); + auto* k = std::get<2>(arguments); + auto* n = std::get<3>(arguments); + const auto group = std::get<4>(arguments); + state.matmul_pad_calls.push_back({m ? *m : -1, k ? *k : -1, + n ? *n : -1, group}); + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success) { + if (m) *m = PaddedRows(n && *n == 1024 ? "matmul-1024" : "matmul-3072", *m); + if (k) *k += state.matmul_k_delta; + if (n) *n += state.matmul_n_delta; + } + return status; + } else if constexpr (std::is_same_v) { + // (stream, m, desc) in 0.5.0: k / n / group_size are no longer + // passed loose, they come from the weights descriptor, which also + // carries the activation and post-feedforward-norm flags that + // select a different ELF family and therefore a different padding. + auto* m = std::get<1>(arguments); + const auto* desc = std::get<2>(arguments); + state.rows_pad_calls.push_back({"ssmlp", m ? *m : -1, + desc ? desc->k : -1, desc ? desc->n : -1, + desc ? desc->group_size : 0u}); + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && m) + *m = PaddedRows("ssmlp", *m); + return status; + } else if constexpr (std::is_same_v) { + auto* m = std::get<1>(arguments); + auto* desc = std::get<2>(arguments); + state.mha_pad_calls.push_back({m ? *m : -1, desc ? *desc : ryzenai_corelib_flat_mha_bf16_desc{}}); + const auto status = Status(Tag::name); + if (status == ryzenai_corelib_status_success && m) + *m = PaddedRows("mha", *m); + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + auto* desc = std::get<0>(arguments); + auto* components = std::get<1>(arguments); + auto* out = std::get<3>(arguments); + if (out) *out = nullptr; + ObserveCreateConcurrency(); + SimulatePackingWork(state_lock); + if (desc && components) state.weight_creates.push_back({"matmul", desc->k, desc->n, + desc->group_size, std::get<2>(arguments), {components->blocks}}); + if (status == ryzenai_corelib_status_success && out) { + *out = NewObject("matmul_weights"); + auto* object = static_cast(*out); + object->byte_size = PackedSize(desc ? desc->k : 0, desc ? desc->n : 0); + object->fill = FillFor(components ? components->blocks : nullptr); + } + --state.active_weight_creates; + return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + auto* desc = std::get<0>(arguments); + auto* components = std::get<1>(arguments); + auto* out = std::get<3>(arguments); + if (out) *out = nullptr; + ObserveCreateConcurrency(); + SimulatePackingWork(state_lock); + if (desc && components) { + fake_corelib::WeightCreateRecord record{"ssmlp", desc->k, desc->n, + desc->group_size, std::get<2>(arguments), + {components->gate_blocks, components->up_blocks, components->down_blocks}}; + if (components->epsilon) record.epsilon = *static_cast(components->epsilon); + if (components->norm0) record.norm0.assign(static_cast(components->norm0), + static_cast(components->norm0) + desc->k); + if (components->norm1) record.norm1.assign(static_cast(components->norm1), + static_cast(components->norm1) + desc->k); + state.weight_creates.push_back(std::move(record)); + } + if (status == ryzenai_corelib_status_success && out) { + *out = NewObject("ssmlp_weights"); + auto* object = static_cast(*out); + object->byte_size = PackedSize(desc ? desc->k : 0, desc ? desc->n : 0); + object->fill = FillFor(components ? components->gate_blocks : nullptr); + } + --state.active_weight_creates; + return status; + } else if constexpr (std::is_same_v) { + // Two-call protocol: NULL out learns the size, then the caller + // calls again with a buffer of at least that many bytes. + auto* weights = static_cast(std::get<0>(arguments)); + auto* out = std::get<1>(arguments); + const auto out_size = std::get<2>(arguments); + auto* size = std::get<3>(arguments); + const std::size_t packed = weights ? weights->byte_size : 0; + if (size) *size = packed; + const auto status = Status(Tag::name); + if (status != ryzenai_corelib_status_success) return status; + if (out == nullptr) return status; + if (out_size < packed) return ryzenai_corelib_status_failure; + std::memset(out, weights ? weights->fill : 0, packed); + return status; + } else if constexpr (std::is_same_v || + std::is_same_v) { + const auto status = Status(Tag::name); + auto* desc = std::get<0>(arguments); + const char* path = std::get<1>(arguments); + const auto offset = std::get<2>(arguments); + const auto size = std::get<3>(arguments); + auto* out = std::get<4>(arguments); + if (out) *out = nullptr; + const bool matmul = + std::is_same_v; + state.weight_from_file.push_back({matmul ? "matmul" : "ssmlp", + path ? path : "", offset, size}); + // corelib rejects a slice that is not exactly what the descriptor + // packs to, because a truncated blob is still a plausible one. + if (desc && size != PackedSize(desc->k, desc->n)) { + return ryzenai_corelib_status_failure; + } + if (status == ryzenai_corelib_status_success && out) { + *out = NewObject(matmul ? "matmul_weights" : "ssmlp_weights"); + auto* object = static_cast(*out); + object->byte_size = static_cast(size); + } + return status; + } else if constexpr (std::is_same_v) { + state.work_in_flight = false; + return Status(Tag::name); + } else if constexpr (std::is_same_v || + std::is_same_v || + std::is_same_v) { + if (state.statuses.contains("test_observe_dispatch_concurrency")) { + const int active = ++state.active_leases; + int maximum = state.maximum_active_leases.load(); + while (active > maximum && + !state.maximum_active_leases.compare_exchange_weak(maximum, active)) {} + state_lock.unlock(); + std::this_thread::sleep_for(std::chrono::milliseconds(1)); + state_lock.lock(); + --state.active_leases; + } + const auto status = Status(Tag::name); + if (status != ryzenai_corelib_status_success) return status; + fake_corelib::DispatchRecord record{}; + record.thread_id = std::this_thread::get_id(); + record.kind = std::is_same_v ? "matmul" : + std::is_same_v ? "ssmlp" : "mha"; + record.stream = std::get<0>(arguments); + // 0.5.0 removed the row count from every dispatch: M is the leading + // extent of the operand that was bound. Read it back off the input + // so the recorded value still means what the tests assert about it. + if constexpr (std::is_same_v) { + record.input = std::get<1>(arguments); + record.output = std::get<3>(arguments); + } else if constexpr (std::is_same_v) { + record.input = std::get<1>(arguments); + record.output = std::get<5>(arguments); + } else { + record.input = std::get<2>(arguments); + record.position = std::get<4>(arguments); + record.output = std::get<9>(arguments); + } + if (record.input) { + const auto& shape = static_cast(record.input)->shape; + record.rows = shape.empty() ? 0 : shape.front(); + } + if (record.output && static_cast(record.output)->kind == "window") + record.window_offset = static_cast(record.output)->window_offset; + state.dispatches.push_back(record); + if constexpr (std::is_same_v) { + auto* output = static_cast(record.output); + if (output && output->shape == std::vector({1, 200064})) { + EnsureStorage(*output); + const auto value = Bf16(1.0f); + std::memcpy(output->storage->bytes->data() + + output->window_offset * TypeBytes(output->data_type), + &value, sizeof(value)); + } + } + state.work_in_flight = true; + if (state.fail_after_submit == Tag::name) return ryzenai_corelib_status_failure; + return ryzenai_corelib_status_success; + } else if constexpr (std::is_same_v) { + return Status(Tag::name); + } else { + static_assert(kAlwaysFalse, "unhandled fake corelib ABI result"); + } + } +}; + +#define FLM_ASSERT_FAKE_ABI(member, symbol) \ + static_assert(std::is_same_v< \ + decltype(&TypedFake::Invoke), \ + decltype(&::symbol)>); +FLM_CORELIB_FUNCTIONS(FLM_ASSERT_FAKE_ABI) +#undef FLM_ASSERT_FAKE_ABI + +void* FunctionFor(std::string_view name) { +#define FLM_MAP_FAKE_FUNCTION(member, symbol) \ + if (name == #symbol) { \ + return reinterpret_cast( \ + &TypedFake::Invoke); \ + } + FLM_CORELIB_FUNCTIONS(FLM_MAP_FAKE_FUNCTION) +#undef FLM_MAP_FAKE_FUNCTION + return nullptr; +} + +template +void CallAndCollect(Result (*function)(Args...), + std::vector& statuses) { + if constexpr (std::is_same_v) { + statuses.push_back(function(Args{}...)); + } else { + function(Args{}...); + } +} +} // namespace + +namespace fake_corelib { + +State& GetState() { return state; } + +void Reset() { + std::lock_guard lock(state_mutex); + state.version = {RYZENAI_CORELIB_VERSION_MAJOR, + RYZENAI_CORELIB_VERSION_MINOR, + RYZENAI_CORELIB_VERSION_PATCH}; + state.selftest_status = ryzenai_corelib_status_success; + state.default_status = ryzenai_corelib_status_success; + state.has_device_context = true; + state.detail.clear(); + state.status_text = "success"; + state.missing_symbol.clear(); + state.resolution_order.clear(); + state.resolution_counts.clear(); + state.call_counts.clear(); + state.statuses.clear(); + state.lifetime_events.clear(); + state.live_objects = 0; + state.releases = 0; + state.cleanup_calls = 0; + state.active_leases = 0; + state.maximum_active_leases = 0; + state.matmul_pad_calls.clear(); + state.rows_pad_calls.clear(); + state.mha_pad_calls.clear(); + state.pad_multiple = 64; + state.matmul_k_delta = 0; + state.matmul_n_delta = 0; + state.pad_row_overrides.clear(); + state.tensor_creates.clear(); + state.tensor_windows.clear(); + state.weight_creates.clear(); + state.weight_from_file.clear(); + state.dispatches.clear(); + state.tensor_writes.clear(); + state.call_log.clear(); + state.active_weight_creates = 0; + state.maximum_active_weight_creates = 0; + state.work_in_flight = false; + state.fail_after_submit.clear(); +} + +flm::corelib::CorelibApi::Resolver Resolver() { + return [](std::string_view name) -> void* { + std::lock_guard lock(state_mutex); + state.resolution_order.emplace_back(name); + ++state.resolution_counts[std::string(name)]; + if (name == state.missing_symbol) return nullptr; + return FunctionFor(name); + }; +} + +std::vector CallEveryResolvedFunction( + const flm::corelib::CorelibFunctions& functions) { + std::vector statuses; +#define FLM_CALL_FAKE_FUNCTION(member, symbol) CallAndCollect(functions.member, statuses); + FLM_CORELIB_FUNCTIONS(FLM_CALL_FAKE_FUNCTION) +#undef FLM_CALL_FAKE_FUNCTION + return statuses; +} + +void* MakeObject() { + std::lock_guard lock(state_mutex); + return NewObject(); +} + +void EnterLease() { + std::lock_guard lock(state_mutex); + const int active = ++state.active_leases; + int maximum = state.maximum_active_leases.load(); + while (active > maximum && + !state.maximum_active_leases.compare_exchange_weak(maximum, active)) {} +} + +void LeaveLease() { + std::lock_guard lock(state_mutex); + --state.active_leases; + state.lifetime_events.emplace_back("lease_leave"); +} + +} // namespace fake_corelib diff --git a/src/test/phi4_rai/fake_corelib.hpp b/src/test/phi4_rai/fake_corelib.hpp new file mode 100644 index 000000000..c5ee85935 --- /dev/null +++ b/src/test/phi4_rai/fake_corelib.hpp @@ -0,0 +1,150 @@ +#pragma once + +#include "rai/corelib_api.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace fake_corelib { + +struct MatmulPadCall { + std::int64_t m; + std::int64_t k; + std::int64_t n; + std::uint32_t group_size; +}; + +struct RowsPadCall { + std::string helper; + std::int64_t m; + std::int64_t k; + std::int64_t n; + std::uint32_t group_size; +}; + +struct MhaPadCall { + std::int64_t m; + ryzenai_corelib_flat_mha_bf16_desc desc; +}; + +struct TensorCreateRecord { + ryzenai_corelib_data_type data_type; + std::vector shape; + void* object; +}; + +struct TensorWindowRecord { + void* parent; + std::vector shape; + std::size_t offset; + void* object; +}; + +struct WeightCreateRecord { + std::string kind; + std::int64_t k; + std::int64_t n; + std::uint32_t group_size; + std::uint32_t threads; + std::vector pointers; + std::vector norm0; + std::vector norm1; + std::uint16_t epsilon{}; +}; + +struct DispatchRecord { + std::thread::id thread_id; + std::string kind; + void* stream; + void* input; + void* output; + std::int64_t rows; + std::int64_t position; + std::size_t window_offset; +}; + +struct TensorWriteRecord { + void* tensor; + ryzenai_corelib_data_type source_type; + std::size_t count; + std::size_t offset; + bool all_zero; +}; + +/// \brief one ..._weights_create_from_file call +struct WeightFromFileRecord { + std::string kind; + std::string path; + std::uint64_t offset{}; + std::uint64_t size{}; +}; + +struct State { + /// \note Derived from the vendor header, not written out: the runtime + /// check in corelib_api.cpp compares against the same macros, so a + /// literal here would fail every test the day the pin moves. + flm::corelib::CorelibVersion version{RYZENAI_CORELIB_VERSION_MAJOR, + RYZENAI_CORELIB_VERSION_MINOR, + RYZENAI_CORELIB_VERSION_PATCH}; + ryzenai_corelib_status selftest_status{ryzenai_corelib_status_success}; + ryzenai_corelib_status default_status{ryzenai_corelib_status_success}; + /// \brief whether get_device() hands back a device rather than NULL + /// \note Still a bool here even though 0.5.0's entry point returns a + /// pointer: what a test wants to say is "this box has an NPU or it + /// does not", and the fake has no device object worth modelling. + bool has_device_context{true}; + /// \brief the PDI pair the engine opened its stream with + /// \note Required from 0.5.0 and deliberately undefaulted by corelib, so + /// it is worth being able to assert on. + int stream_prefill_pdi{-1}; + int stream_token_pdi{-1}; + std::string detail; + std::string status_text{"success"}; + std::string missing_symbol; + std::vector resolution_order; + std::unordered_map resolution_counts; + std::unordered_map call_counts; + std::unordered_map statuses; + std::vector lifetime_events; + std::atomic live_objects{0}; + std::atomic releases{0}; + std::atomic cleanup_calls{0}; + std::atomic active_leases{0}; + std::atomic maximum_active_leases{0}; + std::vector matmul_pad_calls; + std::vector rows_pad_calls; + std::vector mha_pad_calls; + std::int64_t pad_multiple{64}; + std::int64_t matmul_k_delta{0}; + std::int64_t matmul_n_delta{0}; + std::unordered_map> pad_row_overrides; + std::vector tensor_creates; + std::vector tensor_windows; + std::vector weight_creates; + std::vector weight_from_file; + std::vector dispatches; + std::vector tensor_writes; + std::vector call_log; + std::atomic active_weight_creates{0}; + std::atomic maximum_active_weight_creates{0}; + bool work_in_flight{false}; + std::string fail_after_submit; +}; + +State& GetState(); +void Reset(); +flm::corelib::CorelibApi::Resolver Resolver(); +std::vector CallEveryResolvedFunction( + const flm::corelib::CorelibFunctions& functions); +void* MakeObject(); +void EnterLease(); +void LeaveLease(); + +} // namespace fake_corelib diff --git a/src/test/phi4_rai/gguf_fixture.hpp b/src/test/phi4_rai/gguf_fixture.hpp new file mode 100644 index 000000000..ac70bed13 --- /dev/null +++ b/src/test/phi4_rai/gguf_fixture.hpp @@ -0,0 +1,415 @@ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace gguf_fixture { + +inline constexpr std::uint32_t kF32 = 0; +inline constexpr std::uint32_t kQ8_0 = 8; + +enum class Mutation { + None, + TruncatedString, + TruncatedDirectory, + CountOverflow, + ProductOverflow, + OffsetOverflow, + ZeroAlignment, + NonPowerOfTwoAlignment, + DuplicateName, + OutOfFileRange, + OverlappingRanges, + UnsupportedMetadataType, + DtypeMismatch, + ShapeMismatch, + PayloadLengthMismatch, + MisalignedF32, +}; + +struct ArrayValue { + std::uint32_t element_type; + std::uint64_t count; + std::vector encoded_elements; +}; +using MetadataValue = std::variant; + +struct Tensor { + std::string name; + std::vector logical_shape; + std::uint32_t type; + std::uint64_t offset = 0; + bool explicit_offset = false; +}; + +struct TempFile { + std::filesystem::path path; + TempFile() = default; + explicit TempFile(std::filesystem::path value) : path(std::move(value)) {} + TempFile(const TempFile&) = delete; + TempFile& operator=(const TempFile&) = delete; + TempFile(TempFile&& other) noexcept : path(std::move(other.path)) { + other.path.clear(); + } + TempFile& operator=(TempFile&& other) noexcept { + if (this != &other) { + std::error_code ignored; + if (!path.empty()) std::filesystem::remove(path, ignored); + path = std::move(other.path); + other.path.clear(); + } + return *this; + } + ~TempFile() { + std::error_code ignored; + if (!path.empty()) std::filesystem::remove(path, ignored); + } +}; + +template +void Append(std::vector& out, T value) { + static_assert(std::is_trivially_copyable_v); + const auto bytes = std::bit_cast>(value); + out.insert(out.end(), bytes.begin(), bytes.end()); +} + +inline void AppendString(std::vector& out, const std::string& value) { + Append(out, static_cast(value.size())); + for (const char c : value) out.push_back(static_cast(c)); +} + +inline std::uint64_t TensorBytes(const Tensor& tensor) { + std::uint64_t elements = 1; + for (const auto dimension : tensor.logical_shape) { + if (dimension != 0 && elements > std::numeric_limits::max() / dimension) + throw std::overflow_error("fixture tensor product"); + elements *= dimension; + } + if (tensor.type == kF32) return elements * 4; + if (tensor.type == kQ8_0) { + if (elements % 32 != 0) throw std::runtime_error("fixture Q8_0 divisibility"); + return elements / 32 * 34; + } + return elements; +} + +class Builder { +public: + Builder() { AddContractMetadata(); } + + Builder& Alignment(std::uint32_t alignment) { + alignment_ = alignment; + SetMetadata("general.alignment", alignment); + return *this; + } + + Builder& AddMetadata(std::string key, MetadataValue value) { + metadata_.emplace_back(std::move(key), std::move(value)); + return *this; + } + + Builder& SetMetadata(std::string key, MetadataValue value) { + for (auto& entry : metadata_) { + if (entry.first == key) { + entry.second = std::move(value); + return *this; + } + } + return AddMetadata(std::move(key), std::move(value)); + } + + Builder& RemoveMetadata(const std::string& key) { + std::erase_if(metadata_, [&](const auto& entry) { return entry.first == key; }); + return *this; + } + + Builder& AddTensor(std::string name, std::vector logical_shape, + std::uint32_t type) { + tensors_.push_back({std::move(name), std::move(logical_shape), type}); + return *this; + } + + Builder& AddExactFixtureTensors() { + AddTensor("token_embd.weight", {200064, 3072}, kQ8_0); + AddTensor("output_norm.weight", {3072}, kF32); + AddTensor("blk.0.attn_norm.weight", {3072}, kF32); + AddTensor("blk.0.ffn_norm.weight", {3072}, kF32); + AddTensor("blk.0.attn_qkv.weight", {5120, 3072}, kQ8_0); + AddTensor("blk.0.attn_output.weight", {3072, 3072}, kQ8_0); + AddTensor("blk.0.ffn_up.weight", {16384, 3072}, kQ8_0); + AddTensor("blk.0.ffn_down.weight", {3072, 8192}, kQ8_0); + AddTensor("rope_factors_short.weight", {48}, kF32); + return *this; + } + + Builder& AddFullContractTensors(bool short_rope = true) { + AddTensor("token_embd.weight", {200064, 3072}, kQ8_0); + AddTensor("output_norm.weight", {3072}, kF32); + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto prefix = "blk." + std::to_string(layer); + AddTensor(prefix + ".attn_norm.weight", {3072}, kF32); + AddTensor(prefix + ".ffn_norm.weight", {3072}, kF32); + AddTensor(prefix + ".attn_qkv.weight", {5120, 3072}, kQ8_0); + AddTensor(prefix + ".attn_output.weight", {3072, 3072}, kQ8_0); + AddTensor(prefix + ".ffn_up.weight", {16384, 3072}, kQ8_0); + AddTensor(prefix + ".ffn_down.weight", {3072, 8192}, kQ8_0); + } + if (short_rope) AddTensor("rope_factors_short.weight", {48}, kF32); + return *this; + } + + Builder& MutateTensor(const std::string& name, std::uint32_t type, + std::vector shape) { + auto* tensor = FindTensor(name); + if (!tensor) throw std::runtime_error("fixture tensor not found: " + name); + tensor->type = type; + tensor->logical_shape = std::move(shape); + return *this; + } + + Builder& RemoveTensor(const std::string& name) { + std::erase_if(tensors_, [&](const Tensor& tensor) { return tensor.name == name; }); + return *this; + } + + Builder& TruncateTensorPayload(std::string name) { + truncated_tensor_ = std::move(name); + return *this; + } + + Builder& AddEverySkippableMetadataType() { + AddMetadata("skip.u8", std::uint8_t{1}); + AddMetadata("skip.i8", std::int8_t{-1}); + AddMetadata("skip.u16", std::uint16_t{2}); + AddMetadata("skip.i16", std::int16_t{-2}); + AddMetadata("skip.u32", std::uint32_t{3}); + AddMetadata("skip.i32", std::int32_t{-3}); + AddMetadata("skip.f32", 1.25f); + AddMetadata("skip.bool", true); + AddMetadata("skip.string", std::string("value")); + std::vector strings; + AppendString(strings, "a"); AppendString(strings, "bc"); + AddMetadata("skip.array", ArrayValue{8, 2, std::move(strings)}); + AddMetadata("skip.u64", std::uint64_t{4}); + AddMetadata("skip.i64", std::int64_t{-4}); + AddMetadata("skip.f64", 2.5); + return *this; + } + + Builder& Apply(Mutation mutation) { mutation_ = mutation; return *this; } + + TempFile Write(std::string_view label = "fixture") const { + static std::uint64_t serial = 0; + auto path = std::filesystem::temp_directory_path() / + ("flm_phi4_" + std::string(label) + "_" + + std::to_string(++serial) + ".gguf"); + auto bytes = Encode(); + std::ofstream stream(path, std::ios::binary | std::ios::trunc); + if (!stream) throw std::runtime_error("cannot create fixture"); + stream.write(reinterpret_cast(bytes.prefix.data()), + static_cast(bytes.prefix.size())); + if (bytes.file_size > bytes.prefix.size()) { + stream.seekp(static_cast(bytes.file_size - 1)); + const char zero = 0; + stream.write(&zero, 1); + } + stream.close(); + return TempFile(path); + } + +private: + struct Encoded { std::vector prefix; std::uint64_t file_size; }; + + void AddContractMetadata() { + AddMetadata("general.architecture", std::string("phi3")); + AddMetadata("general.alignment", std::uint32_t{32}); + AddMetadata("phi3.block_count", std::uint32_t{32}); + AddMetadata("phi3.context_length", std::uint32_t{131072}); + AddMetadata("phi3.embedding_length", std::uint32_t{3072}); + AddMetadata("phi3.feed_forward_length", std::uint32_t{8192}); + AddMetadata("phi3.attention.head_count", std::uint32_t{24}); + AddMetadata("phi3.attention.head_count_kv", std::uint32_t{8}); + AddMetadata("phi3.attention.layer_norm_rms_epsilon", 1.0e-5f); + AddMetadata("phi3.rope.dimension_count", std::uint32_t{96}); + AddMetadata("phi3.rope.freq_base", 10000.0f); + AddMetadata("phi3.rope.scaling.attn_factor", 1.0f); + AddMetadata("phi3.rope.scaling.original_context_length", std::uint32_t{4096}); + AddMetadata("tokenizer.ggml.tokens", + ArrayValue{0, 200064, std::vector(200064)}); + AddMetadata("tokenizer.ggml.add_bos_token", false); + AddMetadata("tokenizer.ggml.eos_token_id", std::uint32_t{200020}); + } + + Tensor* FindTensor(const std::string& name) { + const auto it = std::find_if(tensors_.begin(), tensors_.end(), + [&](const Tensor& tensor) { return tensor.name == name; }); + return it == tensors_.end() ? nullptr : &*it; + } + + static std::uint32_t TypeOf(const MetadataValue& value) { + return static_cast(value.index()); + } + + static void EncodeValue(std::vector& out, const MetadataValue& value) { + std::visit([&](const auto& item) { + using T = std::decay_t; + if constexpr (std::is_same_v) AppendString(out, item); + else if constexpr (std::is_same_v) { + Append(out, item.element_type); Append(out, item.count); + out.insert(out.end(), item.encoded_elements.begin(), item.encoded_elements.end()); + } else if constexpr (std::is_same_v) Append(out, std::uint8_t(item)); + else Append(out, item); + }, value); + } + + Encoded Encode() const { + auto metadata = metadata_; + auto tensors = tensors_; + std::uint32_t alignment = alignment_; + if (mutation_ == Mutation::ZeroAlignment || + mutation_ == Mutation::NonPowerOfTwoAlignment) { + alignment = mutation_ == Mutation::ZeroAlignment ? 0 : 24; + for (auto& entry : metadata) + if (entry.first == "general.alignment") entry.second = alignment; + } + if (mutation_ == Mutation::DuplicateName && !tensors.empty()) tensors.push_back(tensors.front()); + if (mutation_ == Mutation::DtypeMismatch && !tensors.empty()) tensors.front().type = kF32; + if (mutation_ == Mutation::ShapeMismatch && !tensors.empty()) tensors.front().logical_shape[0]--; + if (!truncated_tensor_.empty()) { + const auto it = std::find_if(tensors.begin(), tensors.end(), [&](const Tensor& tensor) { + return tensor.name == truncated_tensor_; + }); + if (it == tensors.end()) throw std::runtime_error("fixture tensor not found: " + truncated_tensor_); + Tensor target = std::move(*it); + tensors.erase(it); + tensors.push_back(std::move(target)); + } + if (mutation_ == Mutation::MisalignedF32) { + alignment = 1; + for (auto& entry : metadata) + if (entry.first == "general.alignment") entry.second = std::uint32_t{1}; + } + + std::vector out; + Append(out, std::uint32_t{0x46554747}); Append(out, std::uint32_t{3}); + Append(out, mutation_ == Mutation::CountOverflow ? std::numeric_limits::max() + : static_cast(tensors.size())); + Append(out, static_cast(metadata.size())); + for (const auto& [key, value] : metadata) { + AppendString(out, key); + if (mutation_ == Mutation::UnsupportedMetadataType && key == metadata.front().first) { + Append(out, std::uint32_t{99}); + } else { + Append(out, TypeOf(value)); EncodeValue(out, value); + } + } + std::uint64_t running = 0; + std::vector encoded_offset_positions; + for (std::size_t index = 0; index < tensors.size(); ++index) { + auto& tensor = tensors[index]; + if (mutation_ == Mutation::ProductOverflow && index == 0) + tensor.logical_shape = { + static_cast(std::numeric_limits::max()), 3}; + if (tensor.explicit_offset) running = tensor.offset; + if (alignment != 0 && (alignment & (alignment - 1)) == 0) + running = (running + alignment - 1) & ~(std::uint64_t(alignment) - 1); + tensor.offset = running; + const auto size = mutation_ == Mutation::ProductOverflow && index == 0 + ? 0 : TensorBytes(tensor); + if (mutation_ == Mutation::OverlappingRanges && index == 1) { + tensor.offset = 0; + running += size; + } else if (mutation_ == Mutation::OutOfFileRange && index == 0) + tensor.offset = std::uint64_t{1} << 40; + else if (mutation_ == Mutation::OffsetOverflow && index == 0) + tensor.offset = std::numeric_limits::max() - 31; + else running += size; + AppendString(out, tensor.name); + Append(out, static_cast(tensor.logical_shape.size())); + for (auto it = tensor.logical_shape.rbegin(); it != tensor.logical_shape.rend(); ++it) + Append(out, *it); + Append(out, tensor.type); + encoded_offset_positions.push_back(out.size()); + Append(out, tensor.offset); + } + if (mutation_ == Mutation::TruncatedDirectory && !out.empty()) { + out.pop_back(); return {std::move(out), static_cast(out.size())}; + } + const auto data_start = alignment == 0 ? static_cast(out.size()) + : (static_cast(out.size()) + alignment - 1) & ~(std::uint64_t(alignment) - 1); + if (mutation_ == Mutation::MisalignedF32 && !tensors.empty()) { + const std::uint64_t offset = (1 + alignof(float) - data_start % alignof(float)) % alignof(float); + const auto encoded = std::bit_cast>(offset); + std::copy(encoded.begin(), encoded.end(), out.begin() + encoded_offset_positions.front()); + running = std::max(running, offset + TensorBytes(tensors.front())); + } + out.resize(static_cast(data_start), std::byte{0}); + std::uint64_t file_size = data_start + running; + if ((mutation_ == Mutation::PayloadLengthMismatch || !truncated_tensor_.empty()) && + file_size > data_start) --file_size; + if (mutation_ == Mutation::TruncatedString) { + const auto impossible = std::bit_cast>( + std::numeric_limits::max()); + std::copy(impossible.begin(), impossible.end(), out.begin() + 24); + } + return {std::move(out), file_size}; + } + + std::uint32_t alignment_ = 32; + std::vector> metadata_; + std::vector tensors_; + Mutation mutation_ = Mutation::None; + std::string truncated_tensor_; +}; + +inline nlohmann::json ValidConfig() { + return {{"model_type", "phi3"}, {"num_hidden_layers", 32}, + {"hidden_size", 3072}, {"intermediate_size", 8192}, + {"num_attention_heads", 24}, {"num_key_value_heads", 8}, + {"head_dim", 128}, {"vocab_size", 200064}, + {"rms_norm_eps", 1.0e-5}, {"original_max_position_embeddings", 4096}, + {"eos_token_id", 199999}}; +} + +inline nlohmann::json ValidTokenizer() { + nlohmann::json vocab = nlohmann::json::object(); + for (int id = 0; id < 200019; ++id) vocab["t" + std::to_string(id)] = id; + vocab["<|endoftext|>"] = 199999; + vocab["<|end|>"] = 200020; + return {{"model", {{"vocab", std::move(vocab)}}}, + {"added_tokens", nlohmann::json::array({ + {{"id", 200019}, {"content", "<|assistant|>"}}, + {{"id", 200020}, {"content", "<|end|>"}}, + {{"id", 200021}, {"content", "<|user|>"}}, + {{"id", 200022}, {"content", "<|system|>"}}, + {{"id", 200023}, {"content", "<|tool|>"}}, + {{"id", 200024}, {"content", "<|/tool|>"}}, + {{"id", 200025}, {"content", "<|tool_call|>"}}, + {{"id", 200026}, {"content", "<|/tool_call|>"}}, + {{"id", 200027}, {"content", "<|tool_response|>"}}, + {{"id", 200028}, {"content", "<|tag|>"}}, + {{"id", 200018}, {"content", "<|endofprompt|>"}}, + {{"id", 199999}, {"content", "<|endoftext|>"}}})}}; +} + +inline nlohmann::json ValidTokenizerConfig() { + return {{"add_bos_token", false}, + {"chat_template", "<|user|>{{ message }}<|end|><|assistant|>"}}; +} + +} // namespace gguf_fixture diff --git a/src/test/phi4_rai/run_real_rai_acceptance.ps1 b/src/test/phi4_rai/run_real_rai_acceptance.ps1 new file mode 100644 index 000000000..a553d5283 --- /dev/null +++ b/src/test/phi4_rai/run_real_rai_acceptance.ps1 @@ -0,0 +1,232 @@ +param( + [string]$FlmExe = 'src/build-rai/Release/flm.exe', + [string]$Model = 'phi4-mini-it:4b', + [string]$CorelibDll = 'C:/Users/chiz/work/ryzenai-corelib/install/bin/ryzenai_corelib.dll', + [string]$Output = 'src/build-rai/phi4-gguf-rai-acceptance.json', + [int]$Port = 52625, + [string]$Python = 'python', + # Diagnostics only: skips the 16-minute CLI matrix so the REST phase can be + # iterated on quickly. A record produced this way can never report success. + [switch]$SkipCli +) +$ErrorActionPreference='Stop' +$root=(Resolve-Path (Join-Path $PSScriptRoot '../../..')).Path +$exe=(Resolve-Path (Join-Path $root $FlmExe)).Path +$core=(Resolve-Path $CorelibDll).Path +$outPath=[IO.Path]::GetFullPath((Join-Path $root $Output)) +$outDir=Split-Path $outPath +$modelDir=Join-Path $env:USERPROFILE '.flm/models/phi4-mini-it-rai' +# Inert when flm.exe was built with FLM_ENABLE_RAI (corelib is linked +# in there); still used by the DLL-loading test binaries in this directory. +$env:FLM_RAI_CORELIB_PATH=$core +$env:FLM_CONFIG_PATH=Join-Path $root 'src/model_list.json' +$env:FLM_XCLBIN_PATH=Join-Path $root 'src' +$runtime=Split-Path $core +$env:PATH="$(Join-Path $root 'src/lib/xrt');$(Join-Path $root 'src/lib');$runtime;C:/Users/chiz/.conda/envs/hybrid-llm/Library/bin;C:/Users/chiz/work/hybrid-llm/install/xrt_package/xrt;$env:PATH" +New-Item -ItemType Directory -Force $outDir | Out-Null +$record=[ordered]@{started=(Get-Date).ToString('o');passed=$false;commands=@();host=[ordered]@{};provenance=[ordered]@{};files=@();cli=[ordered]@{};rest=[ordered]@{};performance=[ordered]@{};failures=@()} +# Progress markers go to stdout so a run that stalls can be located from the +# transcript alone; a silent 30-minute stall is indistinguishable from work. +function Mark([string]$m){Write-Host ("[mark] "+(Get-Date).ToString('HH:mm:ss.fff')+" "+$m)} +# ConvertTo-Json cannot be used on the record as a whole. Some of the values it +# holds are live .NET objects whose property graphs loop back on themselves, and +# ConvertTo-Json expands such a graph until -Depth runs out, which allocates tens +# of gigabytes and never returns. This was confirmed on both Windows PowerShell +# 5.1 and PowerShell 7.0.0 (3.4 GB and still climbing when killed) — moving to a +# newer engine does not avoid it, so do not remove this. This emitter walks +# the record itself: it refuses to descend past $script:JsonMaxDepth, and it +# refuses to re-enter an object that is already an ancestor of the current node. +# ConvertTo-Json is still used, but only ever on a single scalar string. +$script:JsonMaxDepth=10 +function JsonScalar($s){return (ConvertTo-Json -InputObject ([string]$s))} +function EmitJson($v,[string]$label,[int]$level,$ancestors){ + if($null -eq $v){return 'null'} + if($v -is [string]){return (JsonScalar $v)} + if($v -is [bool]){if($v){return 'true'}else{return 'false'}} + if($v -is [datetime]){return (JsonScalar $v.ToString('o'))} + if($v -is [double] -or $v -is [single]){if([double]::IsNaN($v)-or[double]::IsInfinity($v)){return 'null'};return (([double]$v).ToString('R',[Globalization.CultureInfo]::InvariantCulture))} + if($v -is [ValueType] -and $v -isnot [char] -and $v -isnot [Enum]){return (([string]$v))} + if($level -ge $script:JsonMaxDepth){return (JsonScalar $v)} + # The ancestor test exists for live .NET objects, whose property graphs loop. + # It deliberately does not apply to a PSCustomObject: ConvertFrom-Json only + # ever builds trees, and every object it produces shares one singleton base + # instance, so testing those would report every nested JSON object as a loop. + $bo=$null;try{$bo=$v.PSObject.BaseObject}catch{} + $next=$ancestors + if($null -ne $bo -and $bo -isnot [System.Management.Automation.PSCustomObject]){ + foreach($a in $ancestors){if([object]::ReferenceEquals($a,$bo)){return (JsonScalar '')}} + # The ancestor list must be built with Add, not with "+". Adding an array + # with "+" splices its elements in, which would put every element of an + # array on the ancestor list and make each of them look like a loop. + $next=New-Object Collections.ArrayList + if($null -ne $ancestors){[void]$next.AddRange($ancestors)} + [void]$next.Add($bo) + } + $parts=New-Object Collections.ArrayList + if($v -is [System.Collections.IDictionary]){ + foreach($k in @($v.Keys)){ + $sw=[Diagnostics.Stopwatch]::StartNew() + [void]$parts.Add((JsonScalar $k)+':'+(EmitJson $v[$k] "$label.$k" ($level+1) $next)) + if($level -lt 2){Mark ("json {0}.{1} in {2:N1}s" -f $label,$k,$sw.Elapsed.TotalSeconds)} + } + return '{'+($parts -join ',')+'}' + } + if($v -is [System.Collections.IEnumerable]){ + foreach($e in $v){[void]$parts.Add((EmitJson $e "$label[]" ($level+1) $next))} + return '['+($parts -join ',')+']' + } + $props=@($v.PSObject.Properties) + if($props.Count -gt 0){ + foreach($p in $props){ + $pv=$null;try{$pv=$p.Value}catch{$pv=""} + [void]$parts.Add((JsonScalar $p.Name)+':'+(EmitJson $pv "$label.$($p.Name)" ($level+1) $next)) + } + return '{'+($parts -join ',')+'}' + } + return (JsonScalar $v) +} +function WriteRecord($rec,[string]$path){ + [IO.File]::WriteAllText($path,"{`r`n",[Text.Encoding]::UTF8) + $first=$true + foreach($k in @($rec.Keys)){ + $sw=[Diagnostics.Stopwatch]::StartNew() + try{$t=EmitJson $rec[$k] $k 1 (New-Object Collections.ArrayList)}catch{$t=JsonScalar ("")} + if(-not$first){[IO.File]::AppendAllText($path,",`r`n",[Text.Encoding]::UTF8)} + $first=$false + [IO.File]::AppendAllText($path,(' "{0}": {1}' -f $k,$t),[Text.Encoding]::UTF8) + Mark ("json section {0} written in {1:N1}s" -f $k,$sw.Elapsed.TotalSeconds) + } + [IO.File]::AppendAllText($path,"`r`n}`r`n",[Text.Encoding]::UTF8) +} +# Piping an ErrorRecord to Out-String yields nothing but a newline under some +# host configurations, which would record a failure with no reason attached. +function ErrText($e){ + $parts=@("$($e.Exception.GetType().FullName): $($e.Exception.Message)") + $rendered=($e|Out-String);if(-not [string]::IsNullOrWhiteSpace($rendered)){$parts+=$rendered.Trim()} + if($e.InvocationInfo -and $e.InvocationInfo.PositionMessage){$parts+=$e.InvocationInfo.PositionMessage.Trim()} + if($e.ScriptStackTrace){$parts+=$e.ScriptStackTrace.Trim()} + return ($parts -join "`n") +} +function Cmd([string]$line,[scriptblock]$body){$start=Get-Date;try{&$body;$ec=$LASTEXITCODE;if($null-eq$ec){$ec=0}}catch{$ec=1;$t=ErrText $_;$record.failures+=$t;Mark ("FAILURE in ${line}: "+$t);throw}finally{$record.commands+=@([ordered]@{command=$line;exit_code=$ec;seconds=((Get-Date)-$start).TotalSeconds})}} +# A non-2xx reply is an error in both engines, but the two expose it +# differently: Windows PowerShell hands back a WebResponse to read a stream +# from, PowerShell 7 hands back an HttpResponseMessage and puts the body in +# ErrorDetails. An expected 400 must not depend on which engine is running. +function Post([string]$path,$body,[int]$TimeoutSec=900){ + try{$r=Invoke-WebRequest -UseBasicParsing -Uri "http://127.0.0.1:$Port$path" -Method Post -ContentType 'application/json' -TimeoutSec $TimeoutSec -Body ($body|ConvertTo-Json -Depth 8 -Compress);return [ordered]@{status=[int]$r.StatusCode;text=$r.Content;json=($r.Content|ConvertFrom-Json)}} + catch{ + $resp=$_.Exception.Response + if($null -eq $resp){throw} + $text=$null + if($_.ErrorDetails -and $_.ErrorDetails.Message){$text=$_.ErrorDetails.Message} + elseif($resp.PSObject.Methods['GetResponseStream']){$text=(New-Object IO.StreamReader($resp.GetResponseStream())).ReadToEnd()} + elseif($resp.Content){$text=$resp.Content.ReadAsStringAsync().GetAwaiter().GetResult()} + $json=$null;if(-not [string]::IsNullOrWhiteSpace($text)){try{$json=$text|ConvertFrom-Json}catch{}} + return [ordered]@{status=[int]$resp.StatusCode;text=$text;json=$json} + } +} +# Two things must never reach curl as inline arguments. A JSON body loses its +# double quotes to PowerShell's native-argument quoting and the server sees a +# malformed object, so every body goes to a file and is read back with "@file". +# A header value containing a space is split into two arguments, so the second +# half is taken as another URL ("Could not resolve host: application"); the +# colon form without a space carries the same meaning and cannot split. +$ContentTypeArg='Content-Type:application/json' +function BodyFile([string]$name,$body){$p=Join-Path $outDir $name;Set-Content -Path $p -Value ($body|ConvertTo-Json -Depth 8 -Compress) -Encoding ASCII -NoNewline;return $p} +function CurlStream([string]$name,[string]$path,$body){$f=BodyFile $name $body;$out=(&curl.exe -sS -N -H $ContentTypeArg -d "@$f" "http://127.0.0.1:$Port$path" 2>&1|Out-String);if($LASTEXITCODE-ne 0){throw "curl failed ($LASTEXITCODE) for ${path}: $out"};if($out-match 'Could not resolve host'){throw "curl argument splitting for ${path}: $out"};return $out} +function CurlBackground([string]$name,[string]$path,$body,[string]$outFile){$f=BodyFile $name $body;return (Start-Process curl.exe -ArgumentList @('-sS','-N','-H',$ContentTypeArg,'-d',"@$f","http://127.0.0.1:$Port$path") -RedirectStandardOutput $outFile -PassThru)} +try{ + $record.host.computer=$env:COMPUTERNAME;$record.host.cpu=(Get-CimInstance Win32_Processor).Name;$record.host.npu=(Get-CimInstance Win32_PnPEntity|Where-Object Name -match 'NPU|Neural').Name;$os=Get-CimInstance Win32_OperatingSystem;$record.host.windows="$($os.Caption) $($os.Version) build $($os.BuildNumber)";$record.host.power=(powercfg /getactivescheme|Out-String).Trim() + $gitRev={param($d) if(-not (Test-Path $d)){return 'unknown'};$keep=$ErrorActionPreference;$ErrorActionPreference='SilentlyContinue';$r=$null;try{$r=(& git -C $d rev-parse HEAD 2>$null | Select-Object -First 1)}catch{$r=$null};$ErrorActionPreference=$keep;$global:LASTEXITCODE=0;if([string]::IsNullOrWhiteSpace($r)){'unknown'}else{([string]$r).Trim()}};$record.provenance.fastflow=(& $gitRev $root);$coreRoot=(Resolve-Path (Join-Path $runtime '..')).Path;$record.provenance.corelib=(& $gitRev $coreRoot);$record.provenance.corelib_abi='0.5.0';$record.provenance.gguf_revision='78eb92a46fc37e6b524df991ed9aca9bc6aa7b80';$record.provenance.tokenizer_revision='cfbefacb99257ffa30c83adab238a50856ac3083';$record.provenance.corelib_sha256=(Get-FileHash $core -Algorithm SHA256).Hash.ToLower() + Cmd "$exe check $Model" {&$exe check $Model|Out-Host;if($LASTEXITCODE-ne 0){throw 'check failed'}} + # The packed-weight cache is written into the model directory, so a second run + # finds two more files there. They are derived, not downloaded: name them and + # leave them out of the comparison rather than loosening it. + $cacheNames=@('phi4-rai-weights.json','phi4-rai-weights.bin') + $names=@('Phi-4-mini-instruct.Q8_0.gguf','tokenizer.json','tokenizer_config.json','config.json');$actual=@(Get-ChildItem $modelDir -File|% Name|Where-Object{$cacheNames-notcontains $_});if((Compare-Object ($names|Sort-Object) ($actual|Sort-Object))){throw 'model directory is not exactly the four downloaded files'};foreach($n in $names){$f=Get-Item (Join-Path $modelDir $n);$record.files+=@([ordered]@{name=$n;bytes=$f.Length;sha256=(Get-FileHash $f.FullName -Algorithm SHA256).Hash.ToLower()})} + $record.weight_cache_present_at_start=@($cacheNames|Where-Object{Test-Path (Join-Path $modelDir $_)}) + $py=@' +from winpty import PtyProcess +import os,sys,time,threading,json,re +exe,model,out=sys.argv[1:4] +def run(cmds,timeout=900): + p=PtyProcess.spawn(f'{exe} run {model}',env=os.environ.copy(),dimensions=(50,200));chunks=[] + def rd(): + while p.isalive(): + try: chunks.append(p.read(8192)) + except: break + threading.Thread(target=rd,daemon=True).start(); end=time.time()+timeout + while time.time() +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +using flm::corelib::CorelibApi; +using flm::corelib::CorelibError; +using flm::corelib::CorelibRuntime; +using flm::corelib::UniqueMatMulWeights; +using flm::corelib::UniqueSsMlpWeights; +using flm::corelib::UniqueStream; +using flm::corelib::UniqueTensor; +using flm::corelib::UniqueTensorWindow; + +void SetCorelibPath(const char* value) { +#ifdef _WIN32 + _putenv_s("FLM_RAI_CORELIB_PATH", value ? value : ""); +#else + if (value) setenv("FLM_RAI_CORELIB_PATH", value, 1); + else unsetenv("FLM_RAI_CORELIB_PATH"); +#endif +} + +std::shared_ptr ValidApi() { + return CorelibApi::ResolveForTest(fake_corelib::Resolver()); +} + +void TestVersionIsResolvedBeforeEveryOtherSymbol() { + fake_corelib::Reset(); + ValidApi(); + const auto& order = fake_corelib::GetState().resolution_order; + TEST_REQUIRE(order.size() == 26); + TEST_REQUIRE(order.front() == "ryzenai_corelib_get_version"); +} + +void TestExactlyThePinnedVersionIsAccepted() { + fake_corelib::Reset(); + const auto api = ValidApi(); + const auto version = api->runtime_version(); + TEST_REQUIRE(version.major == RYZENAI_CORELIB_VERSION_MAJOR); + TEST_REQUIRE(version.minor == RYZENAI_CORELIB_VERSION_MINOR); + TEST_REQUIRE(version.patch == RYZENAI_CORELIB_VERSION_PATCH); +} + +void TestMajorMinorAndPatchMismatchesAreRejectedWithBothVersions() { + for (const auto version : {flm::corelib::CorelibVersion{1, 3, 0}, + flm::corelib::CorelibVersion{0, 4, 0}, + flm::corelib::CorelibVersion{0, 3, 1}}) { + fake_corelib::Reset(); + fake_corelib::GetState().version = version; + const std::string error = RequireThrows([&] { ValidApi(); }); + RequireContains(error, "0.5.0"); + RequireContains(error, std::to_string(version.major) + "." + + std::to_string(version.minor) + "." + + std::to_string(version.patch)); + TEST_REQUIRE(fake_corelib::GetState().resolution_order.size() == 1); + } +} + +void TestEveryRequiredSymbolIsResolvedExactlyOnce() { + fake_corelib::Reset(); + ValidApi(); + TEST_REQUIRE(fake_corelib::GetState().resolution_counts.size() == 26); + for (const auto& [name, count] : fake_corelib::GetState().resolution_counts) { + (void)name; + TEST_REQUIRE(count == 1); + } +} + +void TestEveryResolvedFakeFunctionUsesItsExactAbi() { + fake_corelib::Reset(); + const auto api = ValidApi(); + fake_corelib::GetState().call_counts.clear(); + fake_corelib::GetState().default_status = ryzenai_corelib_status_bad_argument; + fake_corelib::GetState().selftest_status = ryzenai_corelib_status_bad_argument; + const auto statuses = fake_corelib::CallEveryResolvedFunction(api->functions()); + TEST_REQUIRE(statuses.size() == 20); + TEST_REQUIRE(std::all_of(statuses.begin(), statuses.end(), [](auto status) { + return status == ryzenai_corelib_status_bad_argument; + })); + TEST_REQUIRE(fake_corelib::GetState().call_counts.size() == 26); + for (const auto& [name, count] : fake_corelib::GetState().call_counts) { + (void)name; + TEST_REQUIRE(count == 1); + } + + fake_corelib::GetState().statuses["ryzenai_corelib_tensor_write"] = + ryzenai_corelib_status_unsupported; + TEST_REQUIRE(api->functions().tensor_write( + nullptr, ryzenai_corelib_data_type_bf16, nullptr, 0, 0) == + ryzenai_corelib_status_unsupported); + TEST_REQUIRE(fake_corelib::GetState() + .call_counts["ryzenai_corelib_tensor_write"] == 2); +} + +void TestStandaloneRmsNormSymbolsAreNotRequired() { + for (const auto* symbol : { + "ryzenai_corelib_rmsnorm_bf16_weights_create_scale", + "ryzenai_corelib_rmsnorm_bf16_pad_rows", + "ryzenai_corelib_rmsnorm_bf16"}) { + fake_corelib::Reset(); + fake_corelib::GetState().missing_symbol = symbol; + (void)ValidApi(); + TEST_REQUIRE(!fake_corelib::GetState().resolution_counts.contains(symbol)); + } +} + +void TestMissingSymbolNamesTheSymbolAndUnloadsTheDll() { + fake_corelib::Reset(); + fake_corelib::GetState().missing_symbol = "ryzenai_corelib_create_stream"; + std::weak_ptr unloaded; + std::string error; + { + auto module_lifetime = std::make_shared(1); + unloaded = module_lifetime; + auto base = fake_corelib::Resolver(); + CorelibApi::Resolver resolver = + [module_lifetime, base](std::string_view name) { return base(name); }; + module_lifetime.reset(); + error = RequireThrows([&] { CorelibApi::ResolveForTest(std::move(resolver)); }); + } + RequireContains(error, "ryzenai_corelib_create_stream"); + TEST_REQUIRE(unloaded.expired()); +} + +void TestCorelibErrorCopiesStatusCallAndThreadLocalDetail() { + fake_corelib::Reset(); + fake_corelib::GetState().detail = "invalid tensor row count"; + fake_corelib::GetState().status_text = "bad argument"; + const auto api = ValidApi(); + try { + api->Check(ryzenai_corelib_status_bad_argument, "tensor_write"); + TEST_REQUIRE(false); + } catch (const CorelibError& error) { + TEST_REQUIRE(error.status() == ryzenai_corelib_status_bad_argument); + TEST_REQUIRE(error.call() == "tensor_write"); + TEST_REQUIRE(error.detail() == "invalid tensor row count"); + RequireContains(error.what(), "bad argument"); + } +} + +void TestEnvironmentPathMustBeAnAbsoluteDllPath() { + SetCorelibPath("relative/ryzenai_corelib.dll"); + RequireContains(RequireThrows([] { + CorelibApi::ResolveLibraryPath("C:/apps/flm"); + }), + "absolute"); + SetCorelibPath("C:/apps/flm/rai"); + RequireContains(RequireThrows([] { + CorelibApi::ResolveLibraryPath("C:/apps/flm"); + }), + ".dll"); + SetCorelibPath(nullptr); +} + +void TestEnvironmentPathWinsOverExecutableRelativePath() { + SetCorelibPath("C:/corelib/custom.dll"); + TEST_REQUIRE(CorelibApi::ResolveLibraryPath("C:/apps/flm") == + std::filesystem::path("C:/corelib/custom.dll")); + SetCorelibPath(nullptr); +} + +void TestFallbackIsExeDirectoryRaiDllNotCurrentDirectory() { + SetCorelibPath(nullptr); + const auto expected = std::filesystem::absolute( + std::filesystem::path("C:/apps/flm") / "rai" / "ryzenai_corelib.dll"); + TEST_REQUIRE(CorelibApi::ResolveLibraryPath("C:/apps/flm") == expected); +} + +void TestEveryUniqueObjectReleasesExactlyOnceAfterMoves() { + fake_corelib::Reset(); + const auto api = ValidApi(); + { + UniqueTensor first(api, fake_corelib::MakeObject()); + UniqueTensor moved(std::move(first)); + UniqueTensor assigned; + assigned = std::move(moved); + UniqueStream stream(api, fake_corelib::MakeObject()); + UniqueTensorWindow window(api, fake_corelib::MakeObject()); + UniqueMatMulWeights matmul(api, fake_corelib::MakeObject()); + UniqueSsMlpWeights ssmlp(api, fake_corelib::MakeObject()); + TEST_REQUIRE(!first && !moved && assigned); + TEST_REQUIRE(api->live_object_count() == 5); + TEST_REQUIRE(fake_corelib::GetState().releases == 0); + } + TEST_REQUIRE(fake_corelib::GetState().releases == 5); + TEST_REQUIRE(api->live_object_count() == 0); +} + +void TestRuntimeRunsDependencySelftestAndRequiresDeviceContext() { + fake_corelib::Reset(); + fake_corelib::GetState().selftest_status = ryzenai_corelib_status_failure; + RequireContains(RequireThrows([] { + CorelibRuntime::CreateForTest(ValidApi()); + }), + "selftest_dependencies"); + + fake_corelib::Reset(); + fake_corelib::GetState().has_device_context = false; + RequireContains(RequireThrows([] { + CorelibRuntime::CreateForTest(ValidApi()); + }), + "device context"); + + fake_corelib::Reset(); + const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); + TEST_REQUIRE(runtime->api() != nullptr); + CorelibRuntime::ShutdownProcess(); +} + +void TestExecutionLeaseSerializesTwoThreads() { + fake_corelib::Reset(); + const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); + std::atomic ready{0}; + auto worker = [&] { + ++ready; + while (ready.load() != 2) std::this_thread::yield(); + auto lease = runtime->AcquireExecution(); + fake_corelib::EnterLease(); + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + fake_corelib::LeaveLease(); + }; + std::thread first(worker); + std::thread second(worker); + first.join(); + second.join(); + TEST_REQUIRE(fake_corelib::GetState().maximum_active_leases == 1); + CorelibRuntime::ShutdownProcess(); +} + +void TestShutdownReleasesExecutionLockBeforeDestroyingRuntimeOwner() { + fake_corelib::Reset(); + bool destroyed = false; + bool destroyed_while_locked = false; + CorelibRuntime::SetDestructionObserverForTest([&](bool execution_lock_held) { + destroyed = true; + destroyed_while_locked = execution_lock_held; + }); + auto runtime = CorelibRuntime::CreateForTest(ValidApi()); + runtime.reset(); + CorelibRuntime::ShutdownProcess(); + CorelibRuntime::SetDestructionObserverForTest({}); + TEST_REQUIRE(destroyed); + TEST_REQUIRE(!destroyed_while_locked); +} + +void TestCleanupRunsAfterTheLastObjectAndOnlyOnce() { + fake_corelib::Reset(); + const auto runtime = CorelibRuntime::CreateForTest(ValidApi()); + auto object = std::make_unique(runtime->api(), + fake_corelib::MakeObject()); + RequireContains(RequireThrows([] { CorelibRuntime::ShutdownProcess(); }), + "live corelib object"); + TEST_REQUIRE(fake_corelib::GetState().cleanup_calls == 0); + object.reset(); + + std::atomic lease_acquired{false}; + std::thread holder([&] { + auto lease = runtime->AcquireExecution(); + fake_corelib::EnterLease(); + lease_acquired = true; + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + fake_corelib::LeaveLease(); + }); + while (!lease_acquired.load()) std::this_thread::yield(); + CorelibRuntime::ShutdownProcess(); + holder.join(); + CorelibRuntime::ShutdownProcess(); + TEST_REQUIRE(fake_corelib::GetState().cleanup_calls == 1); + TEST_REQUIRE(fake_corelib::GetState().releases == 1); + TEST_REQUIRE(fake_corelib::GetState().lifetime_events == + std::vector({"release", "lease_leave", "cleanup"})); +} +} // namespace + +int main() { +#define RUN_TEST(name) RunTest(&name, #name) + RUN_TEST(TestVersionIsResolvedBeforeEveryOtherSymbol); + RUN_TEST(TestExactlyThePinnedVersionIsAccepted); + RUN_TEST(TestMajorMinorAndPatchMismatchesAreRejectedWithBothVersions); + RUN_TEST(TestEveryRequiredSymbolIsResolvedExactlyOnce); + RUN_TEST(TestEveryResolvedFakeFunctionUsesItsExactAbi); + RUN_TEST(TestStandaloneRmsNormSymbolsAreNotRequired); + RUN_TEST(TestMissingSymbolNamesTheSymbolAndUnloadsTheDll); + RUN_TEST(TestCorelibErrorCopiesStatusCallAndThreadLocalDetail); + RUN_TEST(TestEnvironmentPathMustBeAnAbsoluteDllPath); + RUN_TEST(TestEnvironmentPathWinsOverExecutableRelativePath); + RUN_TEST(TestFallbackIsExeDirectoryRaiDllNotCurrentDirectory); + RUN_TEST(TestEveryUniqueObjectReleasesExactlyOnceAfterMoves); + RUN_TEST(TestRuntimeRunsDependencySelftestAndRequiresDeviceContext); + RUN_TEST(TestExecutionLeaseSerializesTwoThreads); + RUN_TEST(TestShutdownReleasesExecutionLockBeforeDestroyingRuntimeOwner); + RUN_TEST(TestCleanupRunsAfterTheLastObjectAndOnlyOnce); +#undef RUN_TEST + return 0; +} diff --git a/src/test/phi4_rai/test_model_downloader.cpp b/src/test/phi4_rai/test_model_downloader.cpp new file mode 100644 index 000000000..1d4c1e15f --- /dev/null +++ b/src/test/phi4_rai/test_model_downloader.cpp @@ -0,0 +1,412 @@ +#include "download_model.hpp" +#include "model_downloader.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include +#include +#include + +namespace { +namespace fs = std::filesystem; + +// One tag now covers both NPU generations; the catalog picks the artifacts. +constexpr const char* kRaiTag = "phi4-mini-it:4b"; +// The rai build still installs under its own directory and reads its own +// model_info.json record set, so those two names stay distinct from the tag. +constexpr const char* kRaiModelInfoKey = "phi4-mini-it-rai:4b"; +constexpr const char* kRaiDirName = "phi4-mini-it-rai"; +constexpr const char* kUnslothRevision = "78eb92a46fc37e6b524df991ed9aca9bc6aa7b80"; +constexpr const char* kMicrosoftRevision = "cfbefacb99257ffa30c83adab238a50856ac3083"; + +/// \brief the shipped catalog entry as one platform resolves it +/// \param platform "stx" or "aie_next" +/// \param tag the model tag to resolve +/// \note Goes through model_list so these tests exercise the real +/// filter-and-merge path rather than the raw JSON. +nlohmann::json ResolvedModel(const std::string& platform, const char* tag) { + std::string path = FLM_SOURCE_DIR "/model_list.json"; + std::string exe_dir = "."; + model_list models(path, exe_dir, platform); + TEST_REQUIRE(models.is_model_supported(tag)); + return models.get_model_info(tag).second; +} + +nlohmann::json ReadJson(const fs::path& path) { + std::ifstream stream(path); + TEST_REQUIRE(stream.is_open()); + return nlohmann::json::parse(stream); +} + +void Write(const fs::path& path, std::string_view bytes) { + fs::create_directories(path.parent_path()); + std::ofstream stream(path, std::ios::binary | std::ios::trunc); + stream.write(bytes.data(), static_cast(bytes.size())); + TEST_REQUIRE(stream.good()); +} + +std::string Read(const fs::path& path) { + std::ifstream stream(path, std::ios::binary); + return {std::istreambuf_iterator(stream), std::istreambuf_iterator()}; +} + +std::size_t CountOccurrences(std::string_view text, std::string_view needle) { + std::size_t count = 0; + for (std::size_t position = text.find(needle); position != std::string_view::npos; + position = text.find(needle, position + needle.size())) { + ++count; + } + return count; +} + +fs::path TempDirectory(std::string_view name) { + const auto path = fs::temp_directory_path() / ("flm-task5-" + std::string(name)); + std::error_code ignored; + fs::remove_all(path, ignored); + fs::create_directories(path); + return path; +} + +std::string FileUrl(const fs::path& path) { + std::string value = fs::absolute(path).generic_string(); +#ifdef _WIN32 + return "file:///" + value; +#else + return "file://" + value; +#endif +} + +void TestRaiCatalogHasExactlyFourFilesAndExpectedDirectoryName() { + const auto model = ResolvedModel("aie_next", kRaiTag); + const std::vector expected = { + "Phi-4-mini-instruct.Q8_0.gguf", "tokenizer.json", + "tokenizer_config.json", "config.json"}; + TEST_REQUIRE(model.at("name") == kRaiDirName); + TEST_REQUIRE(model.at("model_info_key") == kRaiModelInfoKey); + TEST_REQUIRE(model.at("files").get>() == expected); + TEST_REQUIRE(model.at("size").get() == 4100140571ULL); +} + +void TestGgufUrlContainsUnslothRevisionAndFilename() { + const auto model = ResolvedModel("aie_next", kRaiTag); + const auto source = resolve_file_source(model, "Phi-4-mini-instruct.Q8_0.gguf", false); + TEST_REQUIRE(source.url == std::string("https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF/resolve/") + + kUnslothRevision + "/Phi-4-mini-instruct.Q8_0.gguf?download=true"); +} + +void TestThreeFrontendUrlsContainMicrosoftRevisionAndFilename() { + const auto model = ResolvedModel("aie_next", kRaiTag); + for (const std::string filename : {"tokenizer.json", "tokenizer_config.json", "config.json"}) { + const auto source = resolve_file_source(model, filename, false); + TEST_REQUIRE(source.url == std::string("https://huggingface.co/microsoft/Phi-4-mini-instruct/resolve/") + + kMicrosoftRevision + "/" + filename + "?download=true"); + } +} + +void TestExistingSingleSourceEntryKeepsItsCurrentUrl() { + // The same tag on stx: the aie_next override must not leak onto Strix. + const auto model = ResolvedModel("stx", kRaiTag); + TEST_REQUIRE(model.at("name") == "Phi4-mini-Instruct-NPU2"); + TEST_REQUIRE(!model.contains("file_sources")); + TEST_REQUIRE(!model.contains("model_info_key")); + const auto source = resolve_file_source(model, "config.json", false); + TEST_REQUIRE(source.url == + "https://huggingface.co/FastFlowLM/Phi4-mini-Instruct-NPU2/resolve/main/config.json?download=true"); +} + +void TestUnknownFileSourceKeyAndMissingUrlOrRevisionFail() { + nlohmann::json model = { + {"url", "https://example.invalid/base"}, + {"files", {"config.json"}}, + {"file_sources", {{"unknown.json", {{"url", "https://example.invalid/source"}, + {"revision", std::string(40, 'a')}}}}}}; + RequireContains(RequireThrows([&] { resolve_file_source(model, "config.json", false); }), + "unknown file_sources key"); + + model["file_sources"] = {{"config.json", {{"revision", std::string(40, 'a')}}}}; + RequireContains(RequireThrows([&] { resolve_file_source(model, "config.json", false); }), "url"); + model["file_sources"] = {{"config.json", {{"url", "https://example.invalid/source"}}}}; + RequireContains(RequireThrows([&] { resolve_file_source(model, "config.json", false); }), "revision"); + model["file_sources"] = {{"config.json", {{"url", "https://example.invalid/source"}, + {"revision", "NOT-A-COMMIT"}}}}; + RequireContains(RequireThrows([&] { resolve_file_source(model, "config.json", false); }), "revision"); +} + +void TestActualRaiCatalogTreatsPinnedConfigWithoutFlmVersionAsCompatible() { + const auto root = TempDirectory("actual-catalog-version"); + // Take the merged aie_next entry and re-home it in a temp catalog. It carries no + // supported_platforms any more, so the default (stx) constructor keeps it. + const auto model = ResolvedModel("aie_next", kRaiTag); + const nlohmann::json catalog = { + {"model_path", "models"}, + {"models", {{"phi4-mini-it", {{"4b", model}}}}}}; + const auto catalog_path = root / "model_list.json"; + Write(catalog_path, catalog.dump()); + std::string catalog_string = catalog_path.string(); + std::string root_string = root.string(); + model_list models(catalog_string, root_string); + const auto model_path = root / "models" / kRaiDirName; + for (const auto& filename : model.at("files")) { + Write(model_path / filename.get(), "placeholder"); + } + Write(model_path / "config.json", + R"({"architectures":["Phi3ForCausalLM"],"model_type":"phi3"})"); + + ModelDownloader downloader(models); + TEST_REQUIRE(downloader.is_model_downloaded(kRaiTag, true, true) == + ModelDownloader::ModelStatus::Ready); +} + +void TestModelInfoHasExactSizeAndSha256ForEveryRequiredFile() { + const auto all_info = ReadJson(FLM_SOURCE_DIR "/model_info.json"); + const auto& records = all_info.at(kRaiModelInfoKey); + const std::vector> expected = { + {"Phi-4-mini-instruct.Q8_0.gguf", 4084611040ULL, "26188c6050d525376a88b04514c236c5e28a36730f1e936f2a00314212b7ba42"}, + {"tokenizer.json", 15524095ULL, "382cc235b56c725945e149cc25f191da667c836655efd0857b004320e90e91ea"}, + {"tokenizer_config.json", 2932ULL, "9c9b6bc0c94d95f69f826c41069a3e8b387ac3ced89601d201886e99240ac9db"}, + {"config.json", 2504ULL, "ac65d86061d3d0d704ee2511fd0eb8713ef19eb6eedba17c3080a4165d5b933b"}}; + TEST_REQUIRE(records.size() == expected.size()); + std::uint64_t total = 0; + for (const auto& [path, size, sha256] : expected) { + const auto match = std::find_if(records.begin(), records.end(), [&](const auto& record) { + return record.at("path") == path; + }); + TEST_REQUIRE(match != records.end()); + TEST_REQUIRE(match->at("size").get() == size); + TEST_REQUIRE(match->at("sha256") == sha256); + total += size; + } + TEST_REQUIRE(total == 4100140571ULL); +} + +struct DownloaderFixture { + fs::path root = TempDirectory("ready"); + fs::path catalog_path = root / "model_list.json"; + fs::path info_path = root / "model_info.json"; + std::string catalog_string; + std::string root_string; + model_list models; + + DownloaderFixture() + : catalog_string(catalog_path.string()), root_string(root.string()), models() { + const nlohmann::json catalog = { + {"model_path", "models"}, + {"models", {{"test-model", {{"1b", { + {"name", "test-model"}, {"url", "https://example.invalid/repo"}, + {"file_url", "https://example.invalid/api"}, {"flm_min_version", "1.0.3"}, + {"details", {{"family", "phi4"}}}, + // Present but empty: this is what marks the entry as + // pinned-upstream, and the per-file lookup is expected to fail + // on it rather than the entry silently going unverified. + {"file_sources", nlohmann::json::object()}, + {"files", {"config.json", "a.bin", "b.bin", "c.bin"}} + }}}}}}}; + const nlohmann::json info = {{"test-model:1b", { + {{"path", "config.json"}, {"size", 57}, {"sha256", "b8bfba5e42c4cb0b8660ea39fec6fefafddc42fb6a0b17d472177fb7683b2290"}}, + {{"path", "a.bin"}, {"size", 5}, {"sha256", "8ed3f6ad685b959ead7022518e1af76cd816f8e8ec7ccdda1ed4018e8f2223f8"}}, + {{"path", "b.bin"}, {"size", 4}, {"sha256", "f44e64e75f3948e9f73f8dfa94721c4ce8cbb4f265c4790c702b2d41cfbf2753"}}, + {{"path", "c.bin"}, {"size", 5}, {"sha256", "be9d587defa1f0c09ef49eb17e206983a5f8f8289e4281860bd0ee5a19592c67"}} + }}}; + Write(catalog_path, catalog.dump()); + Write(info_path, info.dump()); +#ifdef _WIN32 + _putenv_s("FLM_MODELINFO_PATH", info_path.string().c_str()); +#else + setenv("FLM_MODELINFO_PATH", info_path.string().c_str(), 1); +#endif + models = model_list(catalog_string, root_string); + } + + fs::path model_path() const { return root / "models" / "test-model"; } + void WriteValidFiles() const { + Write(model_path() / "config.json", + R"({"architectures":["Phi3ForCausalLM"],"model_type":"phi3"})"); + Write(model_path() / "a.bin", "alpha"); + Write(model_path() / "b.bin", "beta"); + Write(model_path() / "c.bin", "gamma"); + } +}; + +void TestModelIsReadyOnlyWhenAllFourFinalFilesValidate() { + DownloaderFixture fixture; + fixture.WriteValidFiles(); + ModelDownloader downloader(fixture.models); + TEST_REQUIRE(downloader.is_model_downloaded("test-model:1b") == ModelDownloader::ModelStatus::Ready); + Write(fixture.model_path() / "b.bin", "BETA"); + TEST_REQUIRE(downloader.is_model_downloaded("test-model:1b") == ModelDownloader::ModelStatus::Missing); +} + +void TestPartFileNeverMakesModelReady() { + DownloaderFixture fixture; + fixture.WriteValidFiles(); + fs::rename(fixture.model_path() / "c.bin", fixture.model_path() / "c.bin.part"); + ModelDownloader downloader(fixture.models); + TEST_REQUIRE(downloader.is_model_downloaded("test-model:1b") == ModelDownloader::ModelStatus::Missing); +} + +void TestLegacyReadyCheckDoesNotHashOrDeleteWeights() { + const auto root = TempDirectory("legacy-ready"); + const auto catalog_path = root / "model_list.json"; + const auto info_path = root / "model_info.json"; + const nlohmann::json catalog = { + {"model_path", "models"}, + {"models", {{"legacy-model", {{"1b", { + {"name", "legacy-model"}, {"url", "https://example.invalid/repo"}, + {"file_url", "https://example.invalid/api"}, {"flm_min_version", "1.0.3"}, + {"files", {"config.json", "model.bin"}} + }}}}}}}; + const nlohmann::json info = {{"legacy-model:1b", { + {{"path", "config.json"}, {"size", 23}, {"oid", std::string(40, '0')}}, + {{"path", "model.bin"}, {"size", 8}, {"oid", std::string(40, '0')}} + }}}; + Write(catalog_path, catalog.dump()); + Write(info_path, info.dump()); +#ifdef _WIN32 + _putenv_s("FLM_MODELINFO_PATH", info_path.string().c_str()); +#else + setenv("FLM_MODELINFO_PATH", info_path.string().c_str(), 1); +#endif + std::string catalog_string = catalog_path.string(); + std::string root_string = root.string(); + model_list models(catalog_string, root_string); + const auto model_path = root / "models" / "legacy-model"; + Write(model_path / "config.json", R"({"flm_version":"1.0.3"})"); + Write(model_path / "model.bin", "bad-data"); + + ModelDownloader downloader(models); + TEST_REQUIRE(downloader.is_model_downloaded("legacy-model:1b") == + ModelDownloader::ModelStatus::Ready); + TEST_REQUIRE(Read(model_path / "model.bin") == "bad-data"); + + Write(model_path / "config.json", R"({"model_type":"legacy"})"); + TEST_REQUIRE(downloader.is_model_downloaded("legacy-model:1b", true, true) == + ModelDownloader::ModelStatus::Outdated); +} + +void TestPullAndCheckRejectModelscopeBeforePinnedReadyStateChecks() { + DownloaderFixture fixture; + ModelDownloader downloader(fixture.models); + std::ostringstream output; + auto* previous = std::cout.rdbuf(output.rdbuf()); + const bool pull_ok = downloader.pull_model("test-model:1b", true); + const bool check_ok = downloader.check_model("test-model:1b", true, true); + std::cout.rdbuf(previous); + + TEST_REQUIRE(!pull_ok); + TEST_REQUIRE(!check_ok); + TEST_REQUIRE(CountOccurrences(output.str(), + "pinned Hugging Face per-file sources are required") == 2); +} + +void TestCheckHashesPinnedFilesExactlyOnce() { + DownloaderFixture fixture; + fixture.WriteValidFiles(); + ModelDownloader downloader(fixture.models); + std::ostringstream output; + auto* previous = std::cout.rdbuf(output.rdbuf()); + const bool ok = downloader.check_model("test-model:1b", false, false); + std::cout.rdbuf(previous); + + TEST_REQUIRE(ok); + TEST_REQUIRE(CountOccurrences(output.str(), "Checking file:") == 4); +} + +void TestStartupStatusDoesNotRehashButCheckStillDoes() { + // Re-hashing the 4 GB GGUF on every launch cost ~28 s, 62% of startup, and + // buys nothing a pull-time verification has not already established. The + // run/serve paths ask for status only; `flm check` remains the full check. + DownloaderFixture fixture; + fixture.WriteValidFiles(); + ModelDownloader downloader(fixture.models); + + std::ostringstream fast; + auto* previous = std::cout.rdbuf(fast.rdbuf()); + const auto fast_status = downloader.is_model_downloaded("test-model:1b", false, true); + std::cout.rdbuf(previous); + TEST_REQUIRE(fast_status == ModelDownloader::ModelStatus::Ready); + TEST_REQUIRE(CountOccurrences(fast.str(), "Checking file:") == 0); + + std::ostringstream full; + previous = std::cout.rdbuf(full.rdbuf()); + const bool ok = downloader.check_model("test-model:1b", false, false); + std::cout.rdbuf(previous); + TEST_REQUIRE(ok); + TEST_REQUIRE(CountOccurrences(full.str(), "Checking file:") == 4); +} + +download_utils::DownloadRequest Request(const fs::path& source, const fs::path& destination, + std::uint64_t size, std::string hash) { + return {FileUrl(source), destination, size, download_utils::HashAlgorithm::Sha256, std::move(hash)}; +} + +void TestResumeAppendsToPartThenAtomicallyPromotes() { + const auto root = TempDirectory("resume"); + const auto source = root / "source.bin"; + const auto destination = root / "destination.bin"; + Write(source, "abcdefgh"); + Write(destination.string() + ".part", "abcd"); + TEST_REQUIRE(download_utils::download_file_atomic( + Request(source, destination, 8, "9c56cc51b374c3ba189210d5b6d4bf57790d351c96c47c02190ecf1e430635ab"))); + TEST_REQUIRE(Read(destination) == "abcdefgh"); + TEST_REQUIRE(!fs::exists(destination.string() + ".part")); +} + +void TestWrongSizeOrHashNeverReplacesAValidFinalFile() { + const auto root = TempDirectory("wrong"); + const auto source = root / "source.bin"; + const auto destination = root / "destination.bin"; + Write(source, "ABCDEFGH"); + Write(destination, "abcdefgh"); + TEST_REQUIRE(!download_utils::download_file_atomic( + Request(source, destination, 8, "9c56cc51b374c3ba189210d5b6d4bf57790d351c96c47c02190ecf1e430635ab"))); + TEST_REQUIRE(Read(destination) == "abcdefgh"); + TEST_REQUIRE(!fs::exists(destination.string() + ".part")); +} + +void TestInterruptedTransferKeepsPartForNextResume() { + const auto root = TempDirectory("interrupted"); + const auto destination = root / "destination.bin"; + Write(destination.string() + ".part", "abcd"); + auto request = Request(root / "missing.bin", destination, 8, + "9c56cc51b374c3ba189210d5b6d4bf57790d351c96c47c02190ecf1e430635ab"); + TEST_REQUIRE(!download_utils::download_file_atomic(request)); + TEST_REQUIRE(Read(destination.string() + ".part") == "abcd"); + TEST_REQUIRE(!fs::exists(destination)); +} + +void TestSuccessfulForceDownloadAtomicallyReplacesFinalFile() { + const auto root = TempDirectory("replace"); + const auto source = root / "source.bin"; + const auto destination = root / "destination.bin"; + Write(source, "ABCDEFGH"); + Write(destination, "abcdefgh"); + TEST_REQUIRE(download_utils::download_file_atomic( + Request(source, destination, 8, "9ac2197d9258257b1ae8463e4214e4cd0a578bc1517f2415928b91be4283fc48"))); + TEST_REQUIRE(Read(destination) == "ABCDEFGH"); + TEST_REQUIRE(!fs::exists(destination.string() + ".part")); +} +} // namespace + +int main() { + RunTest(TestRaiCatalogHasExactlyFourFilesAndExpectedDirectoryName, "rai catalog"); + RunTest(TestGgufUrlContainsUnslothRevisionAndFilename, "GGUF URL"); + RunTest(TestThreeFrontendUrlsContainMicrosoftRevisionAndFilename, "frontend URLs"); + RunTest(TestExistingSingleSourceEntryKeepsItsCurrentUrl, "legacy URL"); + RunTest(TestUnknownFileSourceKeyAndMissingUrlOrRevisionFail, "source validation"); + RunTest(TestActualRaiCatalogTreatsPinnedConfigWithoutFlmVersionAsCompatible, + "rai pinned config compatibility"); + RunTest(TestModelInfoHasExactSizeAndSha256ForEveryRequiredFile, "model metadata"); + RunTest(TestModelIsReadyOnlyWhenAllFourFinalFilesValidate, "ready integrity"); + RunTest(TestPartFileNeverMakesModelReady, "part is not ready"); + RunTest(TestLegacyReadyCheckDoesNotHashOrDeleteWeights, "legacy ready behavior"); + RunTest(TestPullAndCheckRejectModelscopeBeforePinnedReadyStateChecks, + "modelscope rejection"); + RunTest(TestCheckHashesPinnedFilesExactlyOnce, "single check verification"); + RunTest(TestStartupStatusDoesNotRehashButCheckStillDoes, "startup status skips rehash"); + RunTest(TestResumeAppendsToPartThenAtomicallyPromotes, "resume and promote"); + RunTest(TestWrongSizeOrHashNeverReplacesAValidFinalFile, "invalid transfer isolation"); + RunTest(TestInterruptedTransferKeepsPartForNextResume, "interrupted transfer"); + RunTest(TestSuccessfulForceDownloadAtomicallyReplacesFinalFile, "atomic replacement"); +} diff --git a/src/test/phi4_rai/test_phi4_engine.cpp b/src/test/phi4_rai/test_phi4_engine.cpp new file mode 100644 index 000000000..55bc3cb7b --- /dev/null +++ b/src/test/phi4_rai/test_phi4_engine.cpp @@ -0,0 +1,742 @@ +#include "models/phi4/rai/phi4_rai.hpp" +#include "models/phi4/rai/phi4_rai_constants.hpp" +#include "models/phi4/rai/phi4_rai_host.hpp" +#include "models/phi4/rai/phi4_rai_weight_cache.hpp" +#include +#include +#include +#include "fake_corelib.hpp" +#include "gguf_fixture.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +using flm::corelib::CorelibApi; +using flm::corelib::CorelibRuntime; +using flm::phi4::Phi4GgufPackage; +using flm::phi4::phi4_rai; + +const std::filesystem::path& FullPackagePath() { + static auto file = gguf_fixture::Builder().AddFullContractTensors(false).Write("engine"); + return file.path; +} + +/// \brief scoped FLM_RAI_WEIGHT_CACHE, restored on the way out +struct ScopedWeightCache { + std::string previous; + bool had_previous{}; + explicit ScopedWeightCache(const std::string& value) { + if (const char* existing = std::getenv("FLM_RAI_WEIGHT_CACHE")) { + previous = existing; had_previous = true; + } + _putenv_s("FLM_RAI_WEIGHT_CACHE", value.c_str()); + } + ~ScopedWeightCache() { + if (had_previous) _putenv_s("FLM_RAI_WEIGHT_CACHE", previous.c_str()); + else _putenv_s("FLM_RAI_WEIGHT_CACHE", ""); + } +}; + +struct Harness { + std::shared_ptr runtime; + std::shared_ptr package; + std::unique_ptr engine; + // Off unless a test asks for it: a cache written by one test would + // otherwise make every later one load instead of pack, and the packing + // assertions would silently stop testing anything. + ScopedWeightCache no_cache{"0"}; + + explicit Harness(std::function configure = {}) { + fake_corelib::Reset(); + if (configure) configure(fake_corelib::GetState()); + runtime = CorelibRuntime::CreateForTest( + CorelibApi::ResolveForTest(fake_corelib::Resolver())); + package = Phi4GgufPackage::Open(FullPackagePath()); + engine = std::make_unique(LM_Config{}, package, runtime); + } + ~Harness() { + engine.reset(); + package.reset(); + runtime.reset(); + CorelibRuntime::ShutdownProcess(); + } +}; + +void TestEngineCreatesOneStreamAndPersistentHelperSizedTensors() { + Harness h; + const auto& state = fake_corelib::GetState(); + TEST_REQUIRE(state.call_counts.at("ryzenai_corelib_create_stream") == 1); + TEST_REQUIRE(state.tensor_creates.size() == 74); + TEST_REQUIRE(state.tensor_creates[0].shape == std::vector({4096, 3072})); + TEST_REQUIRE(state.tensor_creates[3].shape == std::vector({4096, 3072})); + TEST_REQUIRE(state.tensor_creates[4].shape == std::vector({4096, 1024})); + TEST_REQUIRE(state.tensor_creates[6].shape == std::vector({1, 3072})); + TEST_REQUIRE(state.tensor_creates[7].shape == std::vector({1, 200064})); +} + +void TestEngineAllocatesMaximaAcrossAllRowsAndConsumers() { + Harness h([](auto& state) { + state.pad_row_overrides["matmul-3072"][2048] = 5000; + state.pad_row_overrides["matmul-1024"][2048] = 6000; + state.pad_row_overrides["ssmlp"][2048] = 8000; + state.pad_row_overrides["mha"][2048] = 9000; + }); + const auto& tensors = fake_corelib::GetState().tensor_creates; + TEST_REQUIRE(tensors[0].shape == std::vector({8000, 3072})); + TEST_REQUIRE(tensors[1].shape == std::vector({8000, 3072})); + TEST_REQUIRE(tensors[2].shape == std::vector({8000, 3072})); + TEST_REQUIRE(tensors[3].shape == std::vector({9000, 3072})); + TEST_REQUIRE(tensors[4].shape == std::vector({9000, 1024})); + TEST_REQUIRE(tensors[5].shape == std::vector({9000, 3072})); +} + +/// \brief find the one create whose source pointers are exactly these +/// \note The creates run concurrently, so completion order is not defined. +/// What must hold is that every weight was packed from its own mapped +/// range exactly once, which is what these lookups assert. +const fake_corelib::WeightCreateRecord& CreateFrom( + const std::vector& pointers) { + const fake_corelib::WeightCreateRecord* found = nullptr; + for (const auto& record : fake_corelib::GetState().weight_creates) { + if (record.pointers == pointers) { + TEST_REQUIRE(found == nullptr); + found = &record; + } + } + TEST_REQUIRE(found != nullptr); + return *found; +} + +void TestEngineCreatesExactly129MatmulAnd32SsmlpWeights() { + Harness h; + const auto& records = fake_corelib::GetState().weight_creates; + TEST_REQUIRE(records.size() == 161); + TEST_REQUIRE(std::count_if(records.begin(), records.end(), [](const auto& r) { return r.kind == "matmul"; }) == 129); + TEST_REQUIRE(std::count_if(records.begin(), records.end(), [](const auto& r) { return r.kind == "ssmlp"; }) == 32); + TEST_REQUIRE(std::none_of(records.begin(), records.end(), [](const auto& r) { return r.kind == "rmsnorm"; })); +} + +void TestEveryProjectionUsesQ8RequantizedGroup64WithThreadHint() { + // The parallelism is concurrent creates, not the per-create hint, so the + // hint stays at corelib's "one" and the two do not multiply into an + // oversubscribed machine. + Harness h; + for (const auto& record : fake_corelib::GetState().weight_creates) { + TEST_REQUIRE(record.group_size == 64); + TEST_REQUIRE(record.threads == flm::phi4::kRequantizeThreads); + } + TEST_REQUIRE(fake_corelib::GetState().call_counts["ryzenai_corelib_matmul_bf16_weights_create_gguf"] == 0); + TEST_REQUIRE(fake_corelib::GetState().call_counts["ryzenai_corelib_ssmlp_bf16_weights_create_gguf"] == 0); +} + +void TestWeightCreationRunsConcurrentlyWithinItsBudget() { + Harness h; + const auto peak = fake_corelib::GetState().maximum_active_weight_creates.load(); + // Concurrency is the point, so require that it actually happened -- and + // that it stayed inside the budget rather than spawning 161 threads. + TEST_REQUIRE(peak > 1); + TEST_REQUIRE(peak <= static_cast(flm::phi4::kWeightCreateConcurrency)); + // Order is no longer defined, so what is checked is that the whole set was + // created: every layer's five, plus the LM head. + const auto& records = fake_corelib::GetState().weight_creates; + TEST_REQUIRE(records.size() == 32 * 5 + 1); + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto qkv = h.package->AttentionQkv(layer); + const auto gate_up = h.package->GateUp(layer); + TEST_REQUIRE(CreateFrom({qkv.values[0].bytes.data()}).kind == "matmul"); + TEST_REQUIRE(CreateFrom({qkv.values[1].bytes.data()}).kind == "matmul"); + TEST_REQUIRE(CreateFrom({qkv.values[2].bytes.data()}).kind == "matmul"); + TEST_REQUIRE(CreateFrom({ + h.package->RequireQ8("blk." + std::to_string(layer) + ".attn_output.weight", + std::array{3072, 3072}).bytes.data()}).kind == "matmul"); + TEST_REQUIRE(CreateFrom({ + gate_up.values[0].bytes.data(), gate_up.values[1].bytes.data(), + h.package->RequireQ8("blk." + std::to_string(layer) + ".ffn_down.weight", + std::array{3072, 8192}).bytes.data()}).kind == "ssmlp"); + } + TEST_REQUIRE(CreateFrom({ + h.package->RequireQ8("token_embd.weight", + std::array{200064, 3072}).bytes.data()}).kind == "matmul"); +} + +/// \brief the packed weights survive a round trip through the cache +/// \note The point of the cache is that the second load does not requantize. +/// Packing is what model load is, so "did it pack" is the assertion: +/// zero creates and 161 slices bound from the file. +void TestWeightCacheReplacesPackingOnTheSecondLoad() { + const auto directory = std::filesystem::temp_directory_path() / + "flm-weight-cache-roundtrip"; + std::error_code ignored; + std::filesystem::remove_all(directory, ignored); + std::filesystem::create_directories(directory, ignored); + ScopedWeightCache cache(directory.string()); + + { // first load: packs, and leaves a cache behind + fake_corelib::Reset(); + auto runtime = CorelibRuntime::CreateForTest( + CorelibApi::ResolveForTest(fake_corelib::Resolver())); + auto package = Phi4GgufPackage::Open(FullPackagePath()); + auto engine = std::make_unique(LM_Config{}, package, runtime); + TEST_REQUIRE(fake_corelib::GetState().weight_creates.size() == 161); + TEST_REQUIRE(fake_corelib::GetState().weight_from_file.empty()); + engine.reset(); package.reset(); runtime.reset(); + CorelibRuntime::ShutdownProcess(); + } + TEST_REQUIRE(std::filesystem::exists( + flm::phi4::WeightCacheDataPath(directory))); + + { // second load: binds every weight from the file, packs nothing + fake_corelib::Reset(); + auto runtime = CorelibRuntime::CreateForTest( + CorelibApi::ResolveForTest(fake_corelib::Resolver())); + auto package = Phi4GgufPackage::Open(FullPackagePath()); + auto engine = std::make_unique(LM_Config{}, package, runtime); + const auto& state = fake_corelib::GetState(); + TEST_REQUIRE(state.weight_creates.empty()); + TEST_REQUIRE(state.weight_from_file.size() == 161); + // Slices must be contiguous and in slot order, or an entry would bind + // the bytes of a different weight. + std::uint64_t expected_offset = 0; + for (const auto& record : state.weight_from_file) { + TEST_REQUIRE(record.offset == expected_offset); + TEST_REQUIRE(record.size > 0); + expected_offset += record.size; + } + TEST_REQUIRE(state.weight_from_file[4].kind == "ssmlp"); + TEST_REQUIRE(state.weight_from_file[0].kind == "matmul"); + TEST_REQUIRE(state.weight_from_file.back().kind == "matmul"); + engine.reset(); package.reset(); runtime.reset(); + CorelibRuntime::ShutdownProcess(); + } + std::filesystem::remove_all(directory, ignored); +} + +/// \brief a cache that no longer matches its GGUF is ignored, not used +void TestStaleWeightCacheFallsBackToPacking() { + const auto directory = std::filesystem::temp_directory_path() / + "flm-weight-cache-stale"; + std::error_code ignored; + std::filesystem::remove_all(directory, ignored); + std::filesystem::create_directories(directory, ignored); + ScopedWeightCache cache(directory.string()); + { + fake_corelib::Reset(); + auto runtime = CorelibRuntime::CreateForTest( + CorelibApi::ResolveForTest(fake_corelib::Resolver())); + auto package = Phi4GgufPackage::Open(FullPackagePath()); + auto engine = std::make_unique(LM_Config{}, package, runtime); + engine.reset(); package.reset(); runtime.reset(); + CorelibRuntime::ShutdownProcess(); + } + // Rewrite the index with a key that cannot match: a different corelib. + const auto index_path = directory / "phi4-rai-weights.json"; + auto document = nlohmann::json::parse(std::ifstream(index_path), nullptr, false); + TEST_REQUIRE(!document.is_discarded()); + document["corelib_minor"] = 99; + { std::ofstream out(index_path, std::ios::binary | std::ios::trunc); + out << document.dump(); } + + const auto stale_data = flm::phi4::WeightCacheDataPath(directory); + const auto stale_size = std::filesystem::file_size(stale_data, ignored); + TEST_REQUIRE(stale_size > 0); + + fake_corelib::Reset(); + auto runtime = CorelibRuntime::CreateForTest( + CorelibApi::ResolveForTest(fake_corelib::Resolver())); + auto package = Phi4GgufPackage::Open(FullPackagePath()); + auto engine = std::make_unique(LM_Config{}, package, runtime); + TEST_REQUIRE(fake_corelib::GetState().weight_from_file.empty()); + TEST_REQUIRE(fake_corelib::GetState().weight_creates.size() == 161); + // The stale file must not simply be ignored: it is two gigabytes, and it + // is replaced by a cache written from this load rather than left behind. + const auto fresh_size = std::filesystem::file_size(stale_data, ignored); + TEST_REQUIRE(fresh_size > 0); + const auto index = flm::phi4::ReadWeightCacheIndex( + directory, flm::phi4::MakeWeightCacheKey( + package->Path(), RYZENAI_CORELIB_VERSION_MAJOR, + RYZENAI_CORELIB_VERSION_MINOR, + RYZENAI_CORELIB_VERSION_PATCH, 64, 161)); + TEST_REQUIRE(index.has_value()); + engine.reset(); package.reset(); runtime.reset(); + CorelibRuntime::ShutdownProcess(); + std::filesystem::remove_all(directory, ignored); +} + +/// \brief a cache nobody will use is deleted rather than left occupying disk +void TestStaleWeightCacheIsReclaimed() { + const auto directory = std::filesystem::temp_directory_path() / + "flm-weight-cache-reclaim"; + std::error_code ignored; + std::filesystem::remove_all(directory, ignored); + std::filesystem::create_directories(directory, ignored); + + // A data file and index from "some other model", plus the temporary an + // interrupted write would have left behind. + const auto data = flm::phi4::WeightCacheDataPath(directory); + { std::ofstream out(data, std::ios::binary); out << std::string(4096, 'x'); } + { std::ofstream out(directory / "phi4-rai-weights.json", std::ios::binary); out << "{}"; } + { std::ofstream out(data.string() + ".tmp", std::ios::binary); out << std::string(2048, 'y'); } + + const auto reclaimed = flm::phi4::RemoveWeightCache(directory); + TEST_REQUIRE(reclaimed >= 4096 + 2048); + TEST_REQUIRE(!std::filesystem::exists(data)); + TEST_REQUIRE(!std::filesystem::exists(directory / "phi4-rai-weights.json")); + TEST_REQUIRE(!std::filesystem::exists(data.string() + ".tmp")); + // Removing a cache that is not there is not an error. + TEST_REQUIRE(flm::phi4::RemoveWeightCache(directory) == 0); + std::filesystem::remove_all(directory, ignored); +} + +void TestQkvAndGateUpPointersMatchExactMappedSubranges() { + Harness h; + const auto qkv = h.package->AttentionQkv(0); + const auto gate_up = h.package->GateUp(0); + TEST_REQUIRE(CreateFrom({qkv.values[0].bytes.data()}).k == 3072); + TEST_REQUIRE(CreateFrom({qkv.values[1].bytes.data()}).k == 3072); + TEST_REQUIRE(CreateFrom({qkv.values[2].bytes.data()}).k == 3072); + const auto& mlp = CreateFrom({ + gate_up.values[0].bytes.data(), gate_up.values[1].bytes.data(), + h.package->RequireQ8("blk.0.ffn_down.weight", + std::array{3072, 8192}).bytes.data()}); + TEST_REQUIRE(mlp.pointers[0] == gate_up.values[0].bytes.data()); + TEST_REQUIRE(mlp.pointers[1] == gate_up.values[1].bytes.data()); +} + +void TestValidatedPackageFlowsDirectlyIntoAllRequantizedCreates() { + // CreateFrom matches on the mapped address and requires exactly one hit, so + // finding every weight proves the validated view reached corelib unchanged + // and was not copied, re-derived or packed twice. + Harness h; + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto qkv = h.package->AttentionQkv(layer); + const auto gate_up = h.package->GateUp(layer); + TEST_REQUIRE(CreateFrom({qkv.values[0].bytes.data()}).n == 3072); + TEST_REQUIRE(CreateFrom({qkv.values[1].bytes.data()}).n == 1024); + TEST_REQUIRE(CreateFrom({qkv.values[2].bytes.data()}).n == 1024); + (void)CreateFrom({ + h.package->RequireQ8("blk." + std::to_string(layer) + + ".attn_output.weight", std::array{3072, 3072}) + .bytes.data()}); + (void)CreateFrom({ + gate_up.values[0].bytes.data(), gate_up.values[1].bytes.data(), + h.package->RequireQ8("blk." + std::to_string(layer) + + ".ffn_down.weight", std::array{3072, 8192}) + .bytes.data()}); + } + TEST_REQUIRE(CreateFrom({ + h.package->RequireQ8("token_embd.weight", + std::array{200064, 3072}).bytes.data()}).n == 200064); +} + +void TestFusedNormsAndEpsilonReachCorelibAsBf16() { + Harness h; + const auto expected = flm::phi4::ConvertF32ToBf16(std::array{1.0e-5f})[0]; + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto gate_up = h.package->GateUp(layer); + const auto& record = CreateFrom({ + gate_up.values[0].bytes.data(), gate_up.values[1].bytes.data(), + h.package->RequireQ8("blk." + std::to_string(layer) + + ".ffn_down.weight", std::array{3072, 8192}) + .bytes.data()}); + TEST_REQUIRE(record.epsilon == expected); + TEST_REQUIRE(record.norm0.size() == 3072); + TEST_REQUIRE(record.norm1.size() == 3072); + } +} + +void TestEmbeddingMappingOutlivesAllLazyRowReads() { + fake_corelib::Reset(); + auto runtime = CorelibRuntime::CreateForTest(CorelibApi::ResolveForTest(fake_corelib::Resolver())); + auto package = Phi4GgufPackage::Open(FullPackagePath()); + std::weak_ptr lifetime = package; + auto engine = std::make_unique(LM_Config{}, package, runtime); + package.reset(); + TEST_REQUIRE(!lifetime.expired()); + (void)engine->forward(0); + engine.reset(); + TEST_REQUIRE(lifetime.expired()); + runtime.reset(); + CorelibRuntime::ShutdownProcess(); +} + +void TestNoDeviceObjectExistsWhenPackageValidationFails() { + fake_corelib::Reset(); + auto runtime = CorelibRuntime::CreateForTest(CorelibApi::ResolveForTest(fake_corelib::Resolver())); + auto file = gguf_fixture::Builder().AddExactFixtureTensors().Write("invalid-engine"); + auto package = Phi4GgufPackage::Open(file.path); + RequireContains(RequireThrows([&] { + phi4_rai engine(LM_Config{}, package, runtime); + }), "blk.1"); + TEST_REQUIRE(fake_corelib::GetState().live_objects == 0); + TEST_REQUIRE(fake_corelib::GetState().call_counts["ryzenai_corelib_create_stream"] == 0); + package.reset(); runtime.reset(); CorelibRuntime::ShutdownProcess(); +} + +void TestPrefillDecodesEmbeddingRowsAndAdvancesPosition() { + Harness h; + std::vector ids{2, 1, 2}; + const auto logits = h.engine->prefill(ids); + TEST_REQUIRE(logits.size() == 200064); + TEST_REQUIRE(h.engine->get_current_context_length() == 3); + TEST_REQUIRE(fake_corelib::GetState().tensor_writes[2].source_type == ryzenai_corelib_data_type_fp32); +} + +void TestDecodeUsesOneRowAndAdvancesPosition() { + Harness h; + (void)h.engine->forward(4); + TEST_REQUIRE(h.engine->get_current_context_length() == 1); + TEST_REQUIRE(fake_corelib::GetState().dispatches.front().rows == 1); +} + +void TestVProjectionWritesWindowAtPositionTimes128() { + Harness h; + h.engine->set_context_length(7); + (void)h.engine->forward(1); + // Since corelib 0.5.0 the engine also opens 2-D views of its operands, to + // bind each at the padded extent its op was planned for. A cache write is + // the only 3-D window, so select on that rather than assuming every window + // is a V write -- what this test is about is where the V projection lands, + // not how many views the dispatch path needs. + std::vector cache_windows; + for (const auto& window : fake_corelib::GetState().tensor_windows) { + if (window.shape.size() == 3) cache_windows.push_back(window); + } + TEST_REQUIRE(cache_windows.size() == 32); + TEST_REQUIRE(cache_windows.front().shape == + std::vector({8, 4089, 128})); + TEST_REQUIRE(cache_windows.front().offset == 7 * 128); + TEST_REQUIRE(fake_corelib::GetState().dispatches[2].window_offset == 7 * 128); +} + +void TestEachLayerOrdersQKVThenMhaThenOThenSsmlpOnOneStream() { + Harness h; + (void)h.engine->forward(1); + const auto& calls = fake_corelib::GetState().dispatches; + TEST_REQUIRE(calls.size() == 193); + const void* stream = calls.front().stream; + TEST_REQUIRE(calls.front().kind == "matmul"); + for (std::size_t layer = 0; layer < 32; ++layer) { + const std::size_t base = layer * 6; + TEST_REQUIRE(calls[base + 0].kind == "matmul"); + TEST_REQUIRE(calls[base + 1].kind == "matmul"); + TEST_REQUIRE(calls[base + 2].kind == "matmul"); + TEST_REQUIRE(calls[base + 3].kind == "mha"); + TEST_REQUIRE(calls[base + 4].kind == "matmul"); + TEST_REQUIRE(calls[base + 5].kind == "ssmlp"); + } + TEST_REQUIRE(std::all_of(calls.begin(), calls.end(), [&](const auto& c) { return c.stream == stream; })); +} + +void TestPrefillStagesTheSameFp32EmbeddingIntoHiddenAndResidual() { + Harness h; + fake_corelib::GetState().tensor_writes.clear(); + std::vector ids{1, 2}; + (void)h.engine->prefill(ids); + const auto& writes = fake_corelib::GetState().tensor_writes; + TEST_REQUIRE(writes.size() >= 2); + TEST_REQUIRE(writes[0].source_type == ryzenai_corelib_data_type_fp32); + TEST_REQUIRE(writes[1].source_type == ryzenai_corelib_data_type_fp32); + TEST_REQUIRE(writes[0].count == writes[1].count); +} + +void TestBuffersAreZeroPaddedBeforeSubmissionForEachRowBucket() { + Harness h([](auto& state) { + state.pad_row_overrides["matmul-1024"][64] = 96; + }); + fake_corelib::GetState().tensor_writes.clear(); + std::vector ids{1, 2}; + (void)h.engine->prefill(ids); + const auto& writes = fake_corelib::GetState().tensor_writes; + TEST_REQUIRE(writes[0].count == 96 * 3072); + TEST_REQUIRE(writes[1].count == 96 * 3072); + TEST_REQUIRE(writes[1].all_zero); +} + +void TestForwardSynchronizesBeforeHostReadAndLmHeadRead() { + Harness h; + fake_corelib::GetState().call_log.clear(); + (void)h.engine->forward(1); + const auto& log = fake_corelib::GetState().call_log; + const auto first_sync = std::find(log.begin(), log.end(), "ryzenai_corelib_stream_synchronize"); + const auto first_read = std::find(log.begin(), log.end(), "ryzenai_corelib_tensor_read"); + TEST_REQUIRE(first_sync < first_read); + const auto lm_submit = std::find(first_read, log.end(), "ryzenai_corelib_matmul_bf16"); + const auto second_sync = std::find(lm_submit, log.end(), "ryzenai_corelib_stream_synchronize"); + const auto logits_read = std::find(second_sync, log.end(), "ryzenai_corelib_tensor_read"); + TEST_REQUIRE(lm_submit < second_sync && second_sync < logits_read); +} + +void TestKVCachesRemainFixedAt8By4096By128() { + Harness h; + const auto& creates = fake_corelib::GetState().tensor_creates; + const auto count = std::count_if(creates.begin(), creates.end(), [](const auto& record) { + return record.shape == std::vector({8, 4096, 128}); + }); + TEST_REQUIRE(count == 64); +} + +void TestPrompt4096IsAcceptedOnlyWithoutARequestedDecodeToken() { + Harness h; + std::vector ids(4096, 0); + (void)h.engine->prefill(ids); + TEST_REQUIRE(h.engine->get_current_context_length() == 4096); + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "capacity"); +} + +void TestTotalDecodeWindowStopsAt4095() { + Harness h; + h.engine->set_context_length(4094); + (void)h.engine->forward(0); + TEST_REQUIRE(h.engine->get_current_context_length() == 4095); + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "4095"); +} + +void TestClearContextResetsLogicalPositionWithoutRecreatingWeights() { + Harness h; + (void)h.engine->forward(0); + const auto creates = fake_corelib::GetState().weight_creates.size(); + h.engine->clear_context(); + TEST_REQUIRE(h.engine->get_current_context_length() == 0); + TEST_REQUIRE(fake_corelib::GetState().weight_creates.size() == creates); +} + +void TestCheckpointRestoreChangesOnlyLogicalPosition() { + Harness h; + (void)h.engine->forward(0); + TEST_REQUIRE(h.engine->checkpoint() == 1); + (void)h.engine->forward(0); + const auto creates = fake_corelib::GetState().weight_creates.size(); + TEST_REQUIRE(h.engine->restore() == 1); + TEST_REQUIRE(h.engine->get_current_context_length() == 1); + TEST_REQUIRE(fake_corelib::GetState().weight_creates.size() == creates); +} + +void TestPreSubmitFailureIsRecoverable() { + Harness h; + fake_corelib::GetState().statuses["ryzenai_corelib_tensor_write"] = ryzenai_corelib_status_bad_argument; + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "tensor_write"); + TEST_REQUIRE(!h.engine->poisoned()); + fake_corelib::GetState().statuses.erase("ryzenai_corelib_tensor_write"); + (void)h.engine->forward(0); +} + +void TestPostSubmitFailureSynchronizesThenPoisonsAndClearsState() { + Harness h; + h.engine->set_context_length(3); + h.engine->checkpoint(); + fake_corelib::GetState().fail_after_submit = "ryzenai_corelib_matmul_bf16"; + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "matmul"); + TEST_REQUIRE(h.engine->poisoned()); + TEST_REQUIRE(!fake_corelib::GetState().work_in_flight); +} + +void TestSynchronizeFailurePoisonsAndClearsState() { + Harness h; + fake_corelib::GetState().statuses["ryzenai_corelib_stream_synchronize"] = ryzenai_corelib_status_failure; + RequireContains(RequireThrows([&] { (void)h.engine->forward(0); }), "synchronize"); + TEST_REQUIRE(h.engine->poisoned()); + TEST_REQUIRE(!fake_corelib::GetState().work_in_flight); +} + +void TestPoisonedInstanceRejectsEveryLaterEntryPoint() { + Harness h; + fake_corelib::GetState().fail_after_submit = "ryzenai_corelib_matmul_bf16"; + (void)RequireThrows([&] { (void)h.engine->forward(0); }); + RequireContains(RequireThrows([&] { h.engine->clear_context(); }), "poisoned"); + RequireContains(RequireThrows([&] { (void)h.engine->get_current_context_length(); }), "poisoned"); + std::vector ids{0}; + RequireContains(RequireThrows([&] { (void)h.engine->prefill(ids); }), "poisoned"); +} + +void TestFakeTensorWindowRetainsAndPropagatesParentStorage() { + fake_corelib::Reset(); + auto api = CorelibApi::ResolveForTest(fake_corelib::Resolver()); + const std::array parent_shape{16}; + void* parent = nullptr; + api->Check(api->functions().create_device_tensor( + ryzenai_corelib_data_type_bf16, parent_shape.data(), parent_shape.size(), &parent), + "create parent"); + const std::array original{11, 22, 33, 44}; + api->Check(api->functions().tensor_write(parent, ryzenai_corelib_data_type_bf16, + original.data(), original.size(), 4), + "write parent"); + const std::array window_shape{4}; + void* window = nullptr; + api->Check(api->functions().create_tensor_window( + parent, window_shape.data(), window_shape.size(), 4, &window), "create window"); + std::array read{}; + api->Check(api->functions().tensor_read(window, ryzenai_corelib_data_type_bf16, + read.data(), read.size(), 0), "read window"); + TEST_REQUIRE(read == original); + const std::array replacement{77, 88}; + api->Check(api->functions().tensor_write(window, ryzenai_corelib_data_type_bf16, + replacement.data(), replacement.size(), 1), + "write window"); + std::array reread{}; + api->Check(api->functions().tensor_read(parent, ryzenai_corelib_data_type_bf16, + reread.data(), reread.size(), 4), "read parent"); + TEST_REQUIRE((reread == std::array{11, 77, 88, 44})); + api->Release(parent); + reread.fill(0); + api->Check(api->functions().tensor_read(window, ryzenai_corelib_data_type_bf16, + reread.data(), reread.size(), 0), "reread retained window"); + TEST_REQUIRE((reread == std::array{11, 77, 88, 44})); + api->Release(window); + TEST_REQUIRE(fake_corelib::GetState().live_objects == 0); +} + +void WriteCacheRow(Harness& h, std::size_t tensor_index, int position, + std::uint16_t base) { + auto& record = fake_corelib::GetState().tensor_creates[tensor_index]; + for (std::size_t head = 0; head < 8; ++head) { + std::array values{}; + values.fill(static_cast(base + head)); + h.runtime->api()->Check(h.runtime->api()->functions().tensor_write( + record.object, ryzenai_corelib_data_type_bf16, values.data(), values.size(), + (head * 4096 + position) * 128), "seed cache row"); + } +} + +void TestGetKCacheGathersHeadMajorPosition() { + Harness h; + WriteCacheRow(h, 10, 7, 100); + const auto result = h.engine->get_k_cache(0, 7); + const auto* bits = reinterpret_cast(result.data()); + for (std::size_t head = 0; head < 8; ++head) + for (std::size_t i = 0; i < 128; ++i) + TEST_REQUIRE(bits[head * 128 + i] == 100 + head); +} + +void TestGetVCacheGathersHeadMajorPosition() { + Harness h; + WriteCacheRow(h, 11, 9, 200); + const auto result = h.engine->get_v_cache(0, 9); + const auto* bits = reinterpret_cast(result.data()); + for (std::size_t head = 0; head < 8; ++head) + for (std::size_t i = 0; i < 128; ++i) + TEST_REQUIRE(bits[head * 128 + i] == 200 + head); +} + +void TestCancellationBoundaryLeavesNoOutstandingFakeWork() { + Harness h; + fake_corelib::GetState().fail_after_submit = "ryzenai_corelib_ssmlp_bf16"; + (void)RequireThrows([&] { (void)h.engine->forward(0); }); + TEST_REQUIRE(!fake_corelib::GetState().work_in_flight); +} + +void TestTwoConcurrentRaiRequestsNeverOverlapDispatch() { + Harness h; + auto second_engine = std::make_unique( + LM_Config{}, h.package, h.runtime); + fake_corelib::GetState().maximum_active_leases = 0; + fake_corelib::GetState().statuses["test_observe_dispatch_concurrency"] = + ryzenai_corelib_status_success; + fake_corelib::GetState().dispatches.clear(); + std::barrier start(3); + std::thread first([&] { start.arrive_and_wait(); (void)h.engine->forward(1); }); + std::thread second([&] { start.arrive_and_wait(); (void)second_engine->forward(2); }); + start.arrive_and_wait(); + first.join(); + second.join(); + + const auto& dispatches = fake_corelib::GetState().dispatches; + TEST_REQUIRE(fake_corelib::GetState().maximum_active_leases == 1); + TEST_REQUIRE(dispatches.size() == 386); + const auto first_request = dispatches.front().thread_id; + TEST_REQUIRE(first_request != dispatches.back().thread_id); + TEST_REQUIRE(std::all_of(dispatches.begin(), dispatches.begin() + 193, + [&](const auto& call) { + return call.thread_id == first_request; + })); + TEST_REQUIRE(std::all_of(dispatches.begin() + 193, dispatches.end(), + [&](const auto& call) { + return call.thread_id != first_request; + })); + + // Prove the fake itself does not serialize or race when the runtime lease is + // intentionally bypassed: the overlap detector must report both calls. + fake_corelib::GetState().dispatches.clear(); + fake_corelib::GetState().maximum_active_leases = 0; + std::barrier unsafe_start(3); + std::atomic unsafe_calls_succeeded{true}; + const auto invoke_without_lease = [&] { + unsafe_start.arrive_and_wait(); + if (h.runtime->api()->functions().matmul( + nullptr, nullptr, nullptr, nullptr) != + ryzenai_corelib_status_success) + unsafe_calls_succeeded = false; + }; + std::thread unsafe_first(invoke_without_lease); + std::thread unsafe_second(invoke_without_lease); + unsafe_start.arrive_and_wait(); + unsafe_first.join(); + unsafe_second.join(); + TEST_REQUIRE(unsafe_calls_succeeded); + TEST_REQUIRE(fake_corelib::GetState().maximum_active_leases == 2); + TEST_REQUIRE(fake_corelib::GetState().dispatches.size() == 2); +} + +void TestTenSequentialLoadsReleaseEveryObjectAndNeverEmitAllZeroLogits() { + for (int cycle = 0; cycle < 10; ++cycle) { + { + Harness h; + const auto logits = h.engine->forward(cycle); + const auto* bits = reinterpret_cast(logits.data()); + TEST_REQUIRE(std::any_of(bits, bits + logits.size(), + [](std::uint16_t value) { return value != 0; })); + } + TEST_REQUIRE(fake_corelib::GetState().live_objects == 0); + } +} +} // namespace + +int main() { +#define RUN_TEST(name) RunTest(&name, #name) + RUN_TEST(TestEngineCreatesOneStreamAndPersistentHelperSizedTensors); + RUN_TEST(TestEngineAllocatesMaximaAcrossAllRowsAndConsumers); + RUN_TEST(TestEngineCreatesExactly129MatmulAnd32SsmlpWeights); + RUN_TEST(TestEveryProjectionUsesQ8RequantizedGroup64WithThreadHint); + RUN_TEST(TestWeightCreationRunsConcurrentlyWithinItsBudget); + RUN_TEST(TestWeightCacheReplacesPackingOnTheSecondLoad); + RUN_TEST(TestStaleWeightCacheFallsBackToPacking); + RUN_TEST(TestStaleWeightCacheIsReclaimed); + RUN_TEST(TestQkvAndGateUpPointersMatchExactMappedSubranges); + RUN_TEST(TestValidatedPackageFlowsDirectlyIntoAllRequantizedCreates); + RUN_TEST(TestFusedNormsAndEpsilonReachCorelibAsBf16); + RUN_TEST(TestEmbeddingMappingOutlivesAllLazyRowReads); + RUN_TEST(TestNoDeviceObjectExistsWhenPackageValidationFails); + RUN_TEST(TestPrefillDecodesEmbeddingRowsAndAdvancesPosition); + RUN_TEST(TestDecodeUsesOneRowAndAdvancesPosition); + RUN_TEST(TestVProjectionWritesWindowAtPositionTimes128); + RUN_TEST(TestEachLayerOrdersQKVThenMhaThenOThenSsmlpOnOneStream); + RUN_TEST(TestPrefillStagesTheSameFp32EmbeddingIntoHiddenAndResidual); + RUN_TEST(TestBuffersAreZeroPaddedBeforeSubmissionForEachRowBucket); + RUN_TEST(TestForwardSynchronizesBeforeHostReadAndLmHeadRead); + RUN_TEST(TestKVCachesRemainFixedAt8By4096By128); + RUN_TEST(TestPrompt4096IsAcceptedOnlyWithoutARequestedDecodeToken); + RUN_TEST(TestTotalDecodeWindowStopsAt4095); + RUN_TEST(TestClearContextResetsLogicalPositionWithoutRecreatingWeights); + RUN_TEST(TestCheckpointRestoreChangesOnlyLogicalPosition); + RUN_TEST(TestPreSubmitFailureIsRecoverable); + RUN_TEST(TestPostSubmitFailureSynchronizesThenPoisonsAndClearsState); + RUN_TEST(TestSynchronizeFailurePoisonsAndClearsState); + RUN_TEST(TestPoisonedInstanceRejectsEveryLaterEntryPoint); + RUN_TEST(TestFakeTensorWindowRetainsAndPropagatesParentStorage); + RUN_TEST(TestGetKCacheGathersHeadMajorPosition); + RUN_TEST(TestGetVCacheGathersHeadMajorPosition); + RUN_TEST(TestCancellationBoundaryLeavesNoOutstandingFakeWork); + RUN_TEST(TestTwoConcurrentRaiRequestsNeverOverlapDispatch); + RUN_TEST(TestTenSequentialLoadsReleaseEveryObjectAndNeverEmitAllZeroLogits); +#undef RUN_TEST +} diff --git a/src/test/phi4_rai/test_phi4_frontend.cpp b/src/test/phi4_rai/test_phi4_frontend.cpp new file mode 100644 index 000000000..81156d484 --- /dev/null +++ b/src/test/phi4_rai/test_phi4_frontend.cpp @@ -0,0 +1,868 @@ +#include "test_support.hpp" +#include "gguf_fixture.hpp" +#if defined(FLM_ENABLE_RAI) +#include "fake_corelib.hpp" +#endif +#include "utils/file_access.hpp" + +#include +#include +#if defined(FLM_ENABLE_RAI) +#include +#include +#include +#endif +#include "server.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/// \brief the builtin set, emptied +/// \note The real one lives in builtin_backends.cpp and names every engine, and +/// with them the prebuilt libraries this suite deliberately does not link. +/// Each test registers the backends it needs instead. +namespace flm::backend { +void register_builtin_backends(BackendRegistry&) {} +} // namespace flm::backend + +namespace { + +std::vector g_encoded_tokens; +std::vector g_samples; +std::vector g_opened_paths; +std::size_t g_sample_index{}; + +class FakeEngine final : public causal_lm { +public: + explicit FakeEngine(std::uint32_t limit) : max_length(limit) {} + + buffer forward(int token) override { + ++forward_calls; + forwarded.push_back(token); + if (forward_delay.count()) std::this_thread::sleep_for(forward_delay); + if (fail_forward) { + poisoned_state = true; + throw std::runtime_error("submitted inference failed"); + } + ++position; + return buffer(1); + } + buffer prefill(std::vector& tokens, void*) override { + ++prefill_calls; + if (fail_prefill) { + poisoned_state = true; + throw std::runtime_error("submitted inference failed"); + } + position += static_cast(tokens.size()); + return buffer(1); + } + void set_context_length(int value) override { position = value; } + void load_weights(Q4NX&) override {} + void update_max_length(std::uint32_t value) override { max_length = value; } + void clear_context() override { + if (poisoned_state) throw std::runtime_error("poisoned"); + position = 0; + } + buffer get_k_cache(int, int) override { return buffer(1); } + buffer get_v_cache(int, int) override { return buffer(1); } + int get_current_context_length() override { return position; } + int checkpoint() override { return position; } + int restore() override { return position; } + + std::uint32_t max_length; + int position{}; + int prefill_calls{}; + int forward_calls{}; + bool fail_prefill{}; + bool fail_forward{}; + bool poisoned_state{}; + std::chrono::microseconds forward_delay{0}; + std::vector forwarded; +}; + +struct FactoryState { + int legacy_calls{}; + int rai_calls{}; + bool throw_for_rai{}; + FakeEngine* engine{}; +#if defined(FLM_ENABLE_RAI) + /// \brief the runtime the corelib stub reports as its detail() + std::shared_ptr runtime; +#endif +} g_factory; + +/// \brief a backend wrapping FakeEngine, with a policy the test dictates +/// \note This is the seam the old engine_factory_for_testing_ hook used to be. +/// Registering it under a real backend id means the frontend, AutoModel +/// and the registry all run their production code paths; only the engine +/// and, for corelib, the loaded library are faked. +class StubBackend final : public flm::backend::ModelBackend { +public: + StubBackend(std::string id, std::uint32_t context_length) + : id_(std::move(id)), + engine_(std::make_unique(context_length)) { + g_factory.engine = engine_.get(); + } + + causal_lm& engine() override { return *engine_; } + std::string id() const override { return id_; } + std::string detail() const override { return detail_; } + std::uint32_t max_decode_length() const override { return decode_limit_; } + bool supports_preemption() const override { return supports_preemption_; } + bool forwards_past_eos() const override { return forwards_past_eos_; } + bool poisoned() const noexcept override { return engine_->poisoned_state; } + std::optional> forced_eos_ids() const override { + return forced_eos_ids_; + } + + std::string detail_; + std::uint32_t decode_limit_{}; + bool supports_preemption_{true}; + bool forwards_past_eos_{true}; + std::optional> forced_eos_ids_; + +private: + std::string id_; + std::unique_ptr engine_; +}; + +/// \brief the FastFlowLM NPU backend, with its engine faked +std::unique_ptr MakeFlmStub( + const flm::backend::BackendContext& context) { + ++g_factory.legacy_calls; + return std::make_unique(std::string(flm::backend::kFlmBackendId), + context.context_length); +} + +#if defined(FLM_ENABLE_RAI) +/// \brief read a JSON file, recording the open for the file-access audit +nlohmann::json ReadObserved(const std::filesystem::path& path) { + flm::file_access::ObserveOpen(path); + std::ifstream input(path, std::ios::binary); + if (!input) throw std::runtime_error("Cannot open " + path.string()); + return nlohmann::json::parse(input); +} + +/// \brief the corelib backend, with its engine and runtime faked +/// \note Everything RaiBackend does before it touches the device is +/// repeated verbatim, so a mismatched package still fails with nothing +/// allocated and the file-access audit still sees the real reads. +std::unique_ptr MakeRaiStub( + const flm::backend::BackendContext& context) { + const std::filesystem::path root(context.model_path); + const auto config = ReadObserved(root / "config.json"); + const auto tokenizer_json = ReadObserved(root / "tokenizer.json"); + // tokenizer_config.json arrives already parsed from the frontend, exactly + // as it does for the real backend, so the audit sees a single open. + if (context.tokenizer_config == nullptr) { + throw std::runtime_error( + "Phi-4 rai backend needs the frontend's tokenizer_config.json"); + } + auto package = flm::phi4::Phi4GgufPackage::Open( + root / "Phi-4-mini-instruct.Q8_0.gguf"); + package->ValidatePhi4Contract(config, tokenizer_json, + *context.tokenizer_config); + + ++g_factory.rai_calls; + if (g_factory.throw_for_rai) throw std::runtime_error("missing corelib"); + + auto backend = std::make_unique( + std::string(flm::backend::kRaiBackendId), context.context_length); + backend->decode_limit_ = flm::phi4::kRaiDecodeLimit; + backend->supports_preemption_ = false; + backend->forwards_past_eos_ = false; + backend->forced_eos_ids_ = std::vector({200020, 199999}); + if (g_factory.runtime) { + backend->detail_ = g_factory.runtime->loaded_library_path().string(); + } + return backend; +} +#endif + +class TempPackage final { +public: + explicit TempPackage(bool valid = true) { + static std::uint64_t serial{}; + path_ = std::filesystem::temp_directory_path() / + ("flm-task4-" + std::to_string(++serial)); + std::filesystem::create_directories(path_); + Write(path_ / "config.json", gguf_fixture::ValidConfig()); + Write(path_ / "tokenizer.json", gguf_fixture::ValidTokenizer()); + auto tokenizer_config = gguf_fixture::ValidTokenizerConfig(); + tokenizer_config["eos_token"] = ""; + tokenizer_config["eos_token_id"] = nlohmann::json::array({200020, 199999}); + Write(path_ / "tokenizer_config.json", tokenizer_config); + auto gguf = gguf_fixture::Builder().AddFullContractTensors(false).Write("frontend"); + std::filesystem::rename(gguf.path, path_ / "Phi-4-mini-instruct.Q8_0.gguf"); + if (!valid) { + auto config = gguf_fixture::ValidConfig(); + config["hidden_size"] = 1; + Write(path_ / "config.json", config); + } + } + ~TempPackage() { + std::error_code ignored; + std::filesystem::remove_all(path_, ignored); + } + const std::filesystem::path& path() const { return path_; } +private: + static void Write(const std::filesystem::path& path, const nlohmann::json& value) { + std::ofstream out(path, std::ios::binary); + if (!out) throw std::runtime_error("cannot write package fixture"); + out << value.dump(); + } + std::filesystem::path path_; +}; + +/// \brief a catalog entry as the frontend sees it +/// \note It names no backend. It cannot: a build links one kernel flow, and the +/// catalog entry has already been filtered to the platform that build +/// targets by the time it reaches here. The tests say which backend they +/// mean by passing it to Load, which is the --backend path. +nlohmann::ordered_json ModelInfo() { + return {{"default_context_length", 4096}, + {"details", {{"family", "phi4"}}}}; +} + +chat_meta_info_t Meta() { + chat_meta_info_t value; + value.max_prefill_len = 64; + return value; +} + +lm_uniform_input_t Input(std::optional budget = std::nullopt) { + lm_uniform_input_t value; + value.prompt = "prompt"; + value.requested_max_new_tokens = budget; + return value; +} + +template +void ExpectRequestError(F&& action, int code, bool cleared, std::string_view text) { + try { action(); } + catch (const ModelRequestError& error) { + TEST_REQUIRE(error.http_code() == code); + TEST_REQUIRE(error.session_cleared() == cleared); + RequireContains(error.what(), text); + return; + } + throw std::runtime_error("expected ModelRequestError"); +} + +} // namespace + +Tokenizer::Tokenizer(const std::string& model_path) { + flm::file_access::ObserveOpen( + std::filesystem::path(model_path) / "tokenizer.json"); + is_doubled_encoded = false; +} +Tokenizer::~Tokenizer() = default; +std::vector Tokenizer::encode(const std::string&) { return g_encoded_tokens; } +std::string Tokenizer::decode(const std::vector&) { return "decoded"; } +std::string Tokenizer::run_time_decoder(int token) { return "t" + std::to_string(token); } +SafeTensors::~SafeTensors() = default; + +Sampler::Sampler(int features, sampler_config& config) + : in_features(features), rep_penalty(config.rep_penalty), + freq_penalty(config.freq_penalty), pre_penalty(config.pre_penalty), + top_k(config.top_k), top_p(config.top_p), min_p(config.min_p), + temperature(config.temperature), total_tokens(0), + freq_penalty_window(config.freq_penalty_window), + rep_penalty_window(config.rep_penalty_window), + repeat_last_n(config.repeat_last_n), + use_optimized_sampling(config.use_optimized_sampling) { + logits.resize(1); counters.resize(1); token_positions.resize(1, -1); +} +void Sampler::reset_penalties() {} +int Sampler::sample(buffer&) { + if (g_sample_index < g_samples.size()) return g_samples[g_sample_index++]; + return 7; +} + +namespace utils { +std::string get_executable_directory() { return std::filesystem::current_path().string(); } +} + +namespace flm::phi4::testing { +/// \brief reads the tokenizer contract Phi4 keeps to itself +/// \note Nothing here writes: the seam the tests drive the model through is the +/// backend registry, not this class. +class Phi4FrontendTestAccess final { +public: + static bool HasLegacyNpu(const Phi4& model) { return model.npu != nullptr; } + static const std::string& EosToken(const Phi4& model) { return model.eos_token; } + static const std::vector& EosTokenIds(const Phi4& model) { return model.eos_token_ids; } + static bool HasBosToken(const Phi4& model) { return model.has_bos_token; } +}; +} // namespace flm::phi4::testing + +namespace { +using flm::phi4::testing::Phi4FrontendTestAccess; + +/// \brief the backend ids this suite drives +/// \note A backend id names a kernel provider, not silicon. These are the shared +/// constants from model_backend.hpp, spelled out here so the suite reads +/// as "run this through that kernel flow". +constexpr const char* kFlm = "flm"; +constexpr const char* kRai = "rai"; + +/// \brief Point the phi4 family's backends at the stubs for one test +/// \note replace_backend, not register_backend: every test re-arms the registry, +/// which is process-wide. The OFF build deliberately registers no corelib +/// backend, which is what makes rejecting its id observable. +struct FactoryScope { + FactoryScope() { + g_factory = {}; + g_opened_paths.clear(); + flm::file_access::SetOpenObserver([](const auto& path) { + g_opened_paths.push_back(path); + }); + auto& registry = flm::backend::BackendRegistry::instance(); + registry.replace_backend("phi4", kFlm, MakeFlmStub); +#if defined(FLM_ENABLE_RAI) + registry.replace_backend("phi4", kRai, MakeRaiStub, + flm::phi4::rai_traits()); +#endif + } + ~FactoryScope() { + flm::file_access::SetOpenObserver({}); +#if defined(FLM_ENABLE_RAI) + g_factory.runtime.reset(); +#endif + } +}; + +/// \brief which backend a loaded model ended up on +bool OnRai(const AutoModel& model) { + return model.backend_id() == kRai; +} +bool OnFlm(const AutoModel& model) { return model.backend_id() == kFlm; } + +std::unique_ptr Load(const TempPackage& package, + nlohmann::ordered_json info, + int context = -1, + bool preemption = false, + flm_rt::device* device = reinterpret_cast(1), + const std::string& backend = kFlm) { + auto model = std::make_unique(device); + model->load_model(package.path().string(), std::move(info), context, + preemption, backend); + return model; +} + +void TestAbsentBackendStillBuildsQ4nxPhi4Npu() { + TempPackage package; + FactoryScope scope; + auto model = Load(package, ModelInfo()); + TEST_REQUIRE(g_factory.legacy_calls == 1); + TEST_REQUIRE(g_factory.rai_calls == 0); + TEST_REQUIRE(OnFlm(*model)); + TEST_REQUIRE(Phi4FrontendTestAccess::HasLegacyNpu(*model)); +} + +void TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib() { + TempPackage package; + FactoryScope scope; + auto model = Load(package, ModelInfo()); + g_encoded_tokens = {1}; + g_samples = {7}; + g_sample_index = 0; + auto meta = Meta(); + auto input = Input(1); + std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + (void)model->generate(meta, 1, output); + TEST_REQUIRE(g_factory.legacy_calls == 1); + TEST_REQUIRE(g_factory.rai_calls == 0); +} + +void TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing() { + TempPackage package; + FactoryScope scope; + g_factory.throw_for_rai = true; + auto model = Load(package, ModelInfo()); + g_encoded_tokens = {1}; + auto meta = Meta(); + auto input = Input(1); + TEST_REQUIRE(model->insert(meta, input)); + TEST_REQUIRE(g_factory.legacy_calls == 1); + TEST_REQUIRE(g_factory.rai_calls == 0); +} + +void TestCorelibRaiGgufBuildsOnlyTheCorelibEngine() { + TempPackage package; + FactoryScope scope; + auto model = Load(package, ModelInfo(), -1, false, nullptr, kRai); + TEST_REQUIRE(g_factory.legacy_calls == 0); + TEST_REQUIRE(g_factory.rai_calls == 1); + TEST_REQUIRE(OnRai(*model)); + TEST_REQUIRE(!Phi4FrontendTestAccess::HasLegacyNpu(*model)); +} + +#if defined(FLM_ENABLE_RAI) +void TestRaiProfileUsesCachedRuntimeDllPathAfterEnvironmentChanges() { + TempPackage package; + FactoryScope scope; + fake_corelib::Reset(); + const auto dll_a = std::filesystem::absolute(package.path() / "runtime-a.dll"); + const auto dll_b = std::filesystem::absolute(package.path() / "runtime-b.dll"); + auto api = flm::corelib::CorelibApi::ResolveForTest( + fake_corelib::Resolver(), dll_a); + auto runtime = flm::corelib::CorelibRuntime::CreateForTest(std::move(api)); + +#ifdef _WIN32 + _putenv_s("FLM_RAI_CORELIB_PATH", dll_b.string().c_str()); +#else + setenv("FLM_RAI_CORELIB_PATH", dll_b.string().c_str(), 1); +#endif + // The backend caches the runtime it was built with; a later environment + // change must not move the path it reports. + g_factory.runtime = runtime; + auto rai = Load(package, ModelInfo(), -1, false, nullptr, kRai); + g_factory.runtime.reset(); + const auto rai_profile = rai->show_profile(); + RequireContains(rai_profile, kRai); + RequireContains(rai_profile, dll_a.string()); + TEST_REQUIRE(rai_profile.find(dll_b.string()) == std::string::npos); + + auto legacy = Load(package, ModelInfo()); + const auto legacy_profile = legacy->show_profile(); + TEST_REQUIRE(legacy_profile.find(kRai) == std::string::npos); + TEST_REQUIRE(legacy_profile.find(dll_a.string()) == std::string::npos); + rai.reset(); + runtime.reset(); + flm::corelib::CorelibRuntime::ShutdownProcess(); +#ifdef _WIN32 + _putenv_s("FLM_RAI_CORELIB_PATH", ""); +#else + unsetenv("FLM_RAI_CORELIB_PATH"); +#endif +} +#endif + +void TestNoManifestOnnxConvertedWeightOrCachePathIsOpened() { + TempPackage package; + FactoryScope scope; + auto model = Load(package, ModelInfo(), -1, false, nullptr, kRai); + TEST_REQUIRE(OnRai(*model)); + std::vector names; + for (const auto& path : g_opened_paths) { + const auto text = path.generic_string(); + TEST_REQUIRE(text.find("manifest") == std::string::npos); + TEST_REQUIRE(text.find("onnx") == std::string::npos); + TEST_REQUIRE(text.find("converted") == std::string::npos); + TEST_REQUIRE(text.find("cache") == std::string::npos); + names.push_back(path.filename().string()); + } + std::sort(names.begin(), names.end()); + TEST_REQUIRE(names == std::vector({ + "Phi-4-mini-instruct.Q8_0.gguf", "config.json", "config.json", + "tokenizer.json", "tokenizer.json", "tokenizer_config.json"})); +} + +void TestUnknownBackendIsAnError() { + TempPackage package; + FactoryScope scope; + // Hardware this build has no engine for is refused by name, before any + // engine is constructed. + const auto message = RequireThrows( + [&] { (void)Load(package, ModelInfo(), -1, false, nullptr, "other"); }); + RequireContains(message, "not compiled into this build"); + RequireContains(message, "other"); + TEST_REQUIRE(g_factory.legacy_calls == 0 && g_factory.rai_calls == 0); +} + +void TestFeatureOffRejectsRaiTagWithoutIncludingCorelibHeaders() { +#if !defined(FLM_ENABLE_RAI) + TempPackage package; + FactoryScope scope; + RequireContains(RequireThrows([&] { + (void)Load(package, ModelInfo(), -1, false, nullptr, kRai); + }), "not compiled into this build"); + TEST_REQUIRE(g_factory.legacy_calls == 0 && g_factory.rai_calls == 0); +#endif +} + +void TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation() { + TempPackage package(false); + FactoryScope scope; + RequireThrows([&] { (void)Load(package, ModelInfo(), -1, false, nullptr, kRai); }); + TEST_REQUIRE(g_factory.rai_calls == 0); +} + +void TestMissingCorelibFailsOnlyWhenRaiModelLoads() { + TempPackage package; + FactoryScope scope; + g_factory.throw_for_rai = true; + RequireContains(RequireThrows([&] { (void)Load(package, ModelInfo(), -1, false, nullptr, kRai); }), "missing corelib"); + TEST_REQUIRE(g_factory.legacy_calls == 0); +} + +void TestRaiSelectionWithMissingDllFailsWithoutChangingBackend() { + TempPackage package; + FactoryScope scope; + Phi4 model(nullptr); + g_factory.throw_for_rai = true; + RequireContains(RequireThrows([&] { + model.load_model(package.path().string(), ModelInfo(), -1, false, kRai); + }), "missing corelib"); + TEST_REQUIRE(model.backend_id().empty()); + TEST_REQUIRE(g_factory.rai_calls == 1); + TEST_REQUIRE(g_factory.legacy_calls == 0); +} + +void TestRaiSelectionCannotReachQ4nxPhi4NpuOrCpuFallback() { + TempPackage package; + FactoryScope scope; + g_factory.throw_for_rai = true; + (void)RequireThrows([&] { + (void)Load(package, ModelInfo(), -1, false, nullptr, kRai); + }); + TEST_REQUIRE(g_factory.rai_calls == 1); + TEST_REQUIRE(g_factory.legacy_calls == 0); +} + +void TestOrdinaryModelLoadsAfterAnRaiRuntimeLoadFailure() { + TempPackage package; + FactoryScope scope; + g_factory.throw_for_rai = true; + RequireThrows([&] { (void)Load(package, ModelInfo(), -1, false, nullptr, kRai); }); + g_factory.throw_for_rai = false; + auto ordinary = Load(package, ModelInfo()); + TEST_REQUIRE(g_factory.legacy_calls == 1); + TEST_REQUIRE(OnFlm(*ordinary)); +} + +void TestPreemptionIsRejectedForTheRaiRoute() { + TempPackage package; + FactoryScope scope; + RequireContains(RequireThrows([&] { (void)Load(package, ModelInfo(), -1, true, nullptr, kRai); }), "preemption"); + TEST_REQUIRE(g_factory.rai_calls == 0); +} + +std::unique_ptr ReadyRai(const TempPackage& package) { + return Load(package, ModelInfo(), -1, false, nullptr, kRai); +} + +void TestRenderedPromptPlusExplicitBudgetMayEqual4095() { + TempPackage package; FactoryScope scope; auto model = ReadyRai(package); + g_encoded_tokens.assign(4000, 1); + auto meta = Meta(); auto input = Input(95); + TEST_REQUIRE(model->insert(meta, input)); +} + +void TestRenderedPromptPlusExplicitBudgetAbove4095Is400() { + TempPackage package; FactoryScope scope; auto model = ReadyRai(package); + g_encoded_tokens.assign(4000, 1); + auto meta = Meta(); auto input = Input(96); + ExpectRequestError([&] { (void)model->insert(meta, input); }, 400, false, "4095"); + TEST_REQUIRE(g_factory.engine->prefill_calls == 0); +} + +void TestOmittedZeroAndNegativeSentinelBudgetsCapAtRemainingWindow() { + for (const auto requested : {std::optional{}, std::optional{0}, std::optional{-1}}) { + TempPackage package; FactoryScope scope; auto model = ReadyRai(package); + g_encoded_tokens.assign(4093, 1); g_samples = {11, 12, 13}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(requested); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + (void)model->generate(meta, 4096, output); + TEST_REQUIRE(meta.generated_tokens == 2); + TEST_REQUIRE(meta.stop_reason == MAX_LENGTH_REACHED); + } + + // /api/chat uses generate_with_prompt and retains 4096 only as the legacy + // loop default; omission must not become an explicit rai budget. + TempPackage package; FactoryScope scope; auto model = ReadyRai(package); + g_encoded_tokens.assign(4093, 1); g_samples = {11, 12, 13}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + (void)model->generate_with_prompt(meta, input, 4096, output); + TEST_REQUIRE(meta.generated_tokens == 2); +} + +void TestCancellationBeforePrefillSubmitsNothing() { + TempPackage package; FactoryScope scope; auto model = ReadyRai(package); + g_encoded_tokens = {1, 2}; auto meta = Meta(); auto input = Input(); + int checks = 0; + TEST_REQUIRE(!model->insert(meta, input, [&] { return ++checks >= 2; })); + TEST_REQUIRE(checks >= 2); + TEST_REQUIRE(g_factory.engine->prefill_calls == 0); + TEST_REQUIRE(meta.stop_reason == CANCEL_DETECTED); +} + +void TestCancellationBetweenDecodeStepsStopsWithCancelReason() { + TempPackage package; FactoryScope scope; auto model = ReadyRai(package); + g_encoded_tokens = {1}; g_samples = {11, 12}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + // Cancel once a decode step has actually happened, rather than after a + // fixed number of polls: how often the loop consults the predicate is its + // own business, but it must not dispatch another forward once cancelled. + (void)model->generate(meta, 10, output, + [&] { return g_factory.engine->forward_calls >= 1; }); + TEST_REQUIRE(meta.stop_reason == CANCEL_DETECTED); + TEST_REQUIRE(g_factory.engine->forward_calls == 1); +} + +void TestCancellationReturnsOnlyAfterSynchronize() { + // Fake calls are synchronous by construction: observing one completed call + // before cancellation proves no work remains outstanding at return. + TestCancellationBetweenDecodeStepsStopsWithCancelReason(); +} + +void TestNonStreamingChatGenerateWithPromptForwardsCancellation() { + TempPackage package; FactoryScope scope; auto model = ReadyRai(package); + g_encoded_tokens = {1, 2}; auto meta = Meta(); auto input = Input(); + std::ostringstream output; + int checks = 0; + AutoModel* endpoint_model = model.get(); + const auto response = endpoint_model->generate_with_prompt( + meta, input, 4096, output, [&] { return ++checks >= 2; }); + TEST_REQUIRE(response.empty()); + TEST_REQUIRE(meta.stop_reason == CANCEL_DETECTED); + TEST_REQUIRE(g_factory.engine->prefill_calls == 0); +} + +void TestLegacyTokenizerContractIsPreserved() { + TempPackage package; FactoryScope scope; auto legacy = Load(package, ModelInfo()); + // Main's legacy Phi-4 frontend intentionally did not pass the textual EOS + // token into minja and retained an empty eos_token string. + TEST_REQUIRE(Phi4FrontendTestAccess::EosToken(*legacy).empty()); + TEST_REQUIRE(Phi4FrontendTestAccess::EosTokenIds(*legacy) == + std::vector({200020, 199999})); + + auto rai = ReadyRai(package); + TEST_REQUIRE(Phi4FrontendTestAccess::EosTokenIds(*rai) == + std::vector({200020, 199999})); + TEST_REQUIRE(!Phi4FrontendTestAccess::HasBosToken(*rai)); +} + +void TestSamePathBackendSwitchForcesLegacyInitialization() { + TempPackage package; FactoryScope scope; + Phi4 model(reinterpret_cast(1)); + model.load_model(package.path().string(), ModelInfo(), -1, false, kRai); + TEST_REQUIRE(OnRai(model)); + TEST_REQUIRE(!Phi4FrontendTestAccess::HasLegacyNpu(model)); + // Named explicitly: the default is now the *build's* platform, which for + // this translation unit is rai, so an argument-less reload would be the + // same backend and would not switch anything. + model.load_model(package.path().string(), ModelInfo(), -1, false, kFlm); + TEST_REQUIRE(OnFlm(model)); + TEST_REQUIRE(Phi4FrontendTestAccess::HasLegacyNpu(model)); + TEST_REQUIRE(g_factory.legacy_calls == 1); +} + +void TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned() { + TempPackage package; FactoryScope scope; auto model = ReadyRai(package); + g_encoded_tokens = {1}; g_samples = {11}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + g_factory.engine->fail_forward = true; + ExpectRequestError([&] { (void)model->generate(meta, 3, output); }, 500, true, "unload/reload"); + TEST_REQUIRE(model->get_current_context_length() == 0); +} + +void TestPoisonedModelReturns500UntilReload() { + TempPackage package; FactoryScope scope; auto model = ReadyRai(package); + g_encoded_tokens = {1}; g_samples = {11}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + g_factory.engine->fail_forward = true; + ExpectRequestError([&] { (void)model->generate(meta, 3, output); }, 500, true, "unload/reload"); + ExpectRequestError([&] { (void)model->insert(meta, input); }, 500, true, "unload/reload"); + auto reloaded = ReadyRai(package); + TEST_REQUIRE(reloaded->insert(meta, input)); +} + +void TestEosSelfTerminatesWithoutAnExtraDecode() { + TempPackage package; FactoryScope scope; auto model = ReadyRai(package); + g_encoded_tokens = {1}; g_samples = {200020}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + (void)model->generate(meta, 10, output); + TEST_REQUIRE(g_factory.engine->forward_calls == 0); + TEST_REQUIRE(meta.stop_reason == EOT_DETECTED); +} + +void TestRaiDecodeTimeAndSpeedAreMeasured() { + // The rai path used to have a decode loop of its own that recorded nothing, so + // profile reported "0 us" and a nan speed. It now shares _shared_generate; + // this keeps the hardware acceptance record's decode throughput honest. + TempPackage package; FactoryScope scope; auto model = ReadyRai(package); + g_encoded_tokens = {1}; g_samples = {11, 12, 13, 200020}; g_sample_index = 0; + auto meta = Meta(); auto input = Input(); std::ostringstream output; + TEST_REQUIRE(model->insert(meta, input)); + g_factory.engine->forward_delay = std::chrono::microseconds(2000); + (void)model->generate(meta, 10, output); + TEST_REQUIRE(g_factory.engine->forward_calls == 3); + TEST_REQUIRE(meta.decoding_duration > 0); + const auto profile = model->show_profile(); + TEST_REQUIRE(profile.find("nan") == std::string::npos); + TEST_REQUIRE(profile.find("inf") == std::string::npos); + TEST_REQUIRE(profile.find("Decoding time: 0 ") == std::string::npos); +} + +void TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics() { + for (const auto raw : {std::optional{}, std::optional{0}, std::optional{-2}, std::optional{17}}) { + const auto expected = raw && *raw > 0 ? raw : std::nullopt; + for (int source = 0; source < 5; ++source) + TEST_REQUIRE(normalize_requested_max_new_tokens(raw) == expected); + } +} + +void TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint() { + TEST_REQUIRE(requires_npu_access("POST", "/v1/completions")); + for (int path = 0; path < 5; ++path) { + int releases = 0; + { + NPURequestCompletionGuard guard([&] { ++releases; }); + if (path == 0) guard.complete(); + else if (path == 1) { guard.complete(); guard.complete(); } + else if (path == 2) { NPURequestCompletionGuard moved(std::move(guard)); } + else if (path == 3) { try { throw std::runtime_error("model"); } catch (...) {} } + else { try { throw 1; } catch (...) {} } + } + TEST_REQUIRE(releases == 1); + } +} + +void TestQueueCompletionReleasesImmediatelyOrDelaysQueuedHandoff() { + constexpr auto cooldown = std::chrono::milliseconds(100); + + NPURequestCoordinator empty; + bool released = false; + const auto empty_start = std::chrono::steady_clock::now(); + empty.complete_current([](auto) { TEST_REQUIRE(false); }, + [&] { released = true; }, cooldown); + const auto empty_elapsed = std::chrono::steady_clock::now() - empty_start; + TEST_REQUIRE(released); + TEST_REQUIRE(empty_elapsed < std::chrono::milliseconds(50)); + + NPURequestCoordinator queued; + bool handed_off = false; + bool released_while_queued = false; + TEST_REQUIRE(queued.try_enqueue([] {})); + const auto queued_start = std::chrono::steady_clock::now(); + queued.complete_current( + [&](auto task) { + handed_off = true; + task(); + }, + [&] { released_while_queued = true; }, cooldown); + const auto queued_elapsed = std::chrono::steady_clock::now() - queued_start; + TEST_REQUIRE(handed_off); + TEST_REQUIRE(!released_while_queued); + TEST_REQUIRE(queued_elapsed >= std::chrono::milliseconds(75)); +} + +void TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable() { + TempPackage package; + FactoryScope scope; + auto model = ReadyRai(package); + NPURequestCoordinator coordinator(3); + bool cancelled = false; + bool capacity_failed = false; + bool queued_request_ran = false; + int completion_callbacks = 0; + int accelerator_releases = 0; + + TEST_REQUIRE(coordinator.try_enqueue([&] { + auto meta = Meta(); + auto input = Input(1); + g_encoded_tokens = {1}; + cancelled = !model->insert(meta, input, [] { return true; }); + })); + TEST_REQUIRE(coordinator.try_enqueue([&] { + auto meta = Meta(); + auto input = Input(1); + g_encoded_tokens.assign(4095, 1); + try { (void)model->insert(meta, input); } + catch (const ModelRequestError& error) { + capacity_failed = error.http_code() == 400; + } + })); + TEST_REQUIRE(coordinator.try_enqueue([&] { + auto meta = Meta(); + auto input = Input(1); + g_encoded_tokens = {1}; + queued_request_ran = model->insert(meta, input); + })); + TEST_REQUIRE(!coordinator.try_enqueue([] {})); + + std::function)> execute; + const auto complete = [&] { + ++completion_callbacks; + coordinator.complete_current(execute, [&] { ++accelerator_releases; }, + std::chrono::milliseconds(0)); + }; + execute = [&](std::function task) { + NPURequestCompletionGuard completion(complete); + task(); + completion.complete(); + completion.complete(); + }; + { + NPURequestCompletionGuard active_request_completion(complete); + active_request_completion.complete(); + active_request_completion.complete(); + } + + TEST_REQUIRE(cancelled); + TEST_REQUIRE(capacity_failed); + TEST_REQUIRE(queued_request_ran); + TEST_REQUIRE(coordinator.empty()); + TEST_REQUIRE(completion_callbacks == 4); + TEST_REQUIRE(accelerator_releases == 1); +} + +} // namespace + +int main() { +#if defined(FLM_ENABLE_RAI) + RunTest(TestAbsentBackendStillBuildsQ4nxPhi4Npu, "TestAbsentBackendStillBuildsQ4nxPhi4Npu"); + RunTest(TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing, "TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing"); + RunTest(TestCorelibRaiGgufBuildsOnlyTheCorelibEngine, "TestCorelibRaiGgufBuildsOnlyTheCorelibEngine"); + RunTest(TestRaiProfileUsesCachedRuntimeDllPathAfterEnvironmentChanges, "TestRaiProfileUsesCachedRuntimeDllPathAfterEnvironmentChanges"); + RunTest(TestNoManifestOnnxConvertedWeightOrCachePathIsOpened, "TestNoManifestOnnxConvertedWeightOrCachePathIsOpened"); + RunTest(TestUnknownBackendIsAnError, "TestUnknownBackendIsAnError"); + RunTest(TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation, "TestInvalidPackageFailsBeforeRuntimeAndDeviceCreation"); + RunTest(TestMissingCorelibFailsOnlyWhenRaiModelLoads, "TestMissingCorelibFailsOnlyWhenRaiModelLoads"); + RunTest(TestRaiSelectionWithMissingDllFailsWithoutChangingBackend, "TestRaiSelectionWithMissingDllFailsWithoutChangingBackend"); + RunTest(TestRaiSelectionCannotReachQ4nxPhi4NpuOrCpuFallback, "TestRaiSelectionCannotReachQ4nxPhi4NpuOrCpuFallback"); + RunTest(TestOrdinaryModelLoadsAfterAnRaiRuntimeLoadFailure, "TestOrdinaryModelLoadsAfterAnRaiRuntimeLoadFailure"); + RunTest(TestPreemptionIsRejectedForTheRaiRoute, "TestPreemptionIsRejectedForTheRaiRoute"); + RunTest(TestRenderedPromptPlusExplicitBudgetMayEqual4095, "TestRenderedPromptPlusExplicitBudgetMayEqual4095"); + RunTest(TestRenderedPromptPlusExplicitBudgetAbove4095Is400, "TestRenderedPromptPlusExplicitBudgetAbove4095Is400"); + RunTest(TestOmittedZeroAndNegativeSentinelBudgetsCapAtRemainingWindow, "TestOmittedZeroAndNegativeSentinelBudgetsCapAtRemainingWindow"); + RunTest(TestCancellationBeforePrefillSubmitsNothing, "TestCancellationBeforePrefillSubmitsNothing"); + RunTest(TestCancellationBetweenDecodeStepsStopsWithCancelReason, "TestCancellationBetweenDecodeStepsStopsWithCancelReason"); + RunTest(TestCancellationReturnsOnlyAfterSynchronize, "TestCancellationReturnsOnlyAfterSynchronize"); + RunTest(TestNonStreamingChatGenerateWithPromptForwardsCancellation, "TestNonStreamingChatGenerateWithPromptForwardsCancellation"); + RunTest(TestLegacyTokenizerContractIsPreserved, "TestLegacyTokenizerContractIsPreserved"); + RunTest(TestSamePathBackendSwitchForcesLegacyInitialization, "TestSamePathBackendSwitchForcesLegacyInitialization"); + RunTest(TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned, "TestPostSubmitErrorReturns500ClearsConversationAndLeavesModelPoisoned"); + RunTest(TestPoisonedModelReturns500UntilReload, "TestPoisonedModelReturns500UntilReload"); + RunTest(TestEosSelfTerminatesWithoutAnExtraDecode, "TestEosSelfTerminatesWithoutAnExtraDecode"); + RunTest(TestRaiDecodeTimeAndSpeedAreMeasured, "TestRaiDecodeTimeAndSpeedAreMeasured"); + RunTest(TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics, "TestCliAndAllFourGenerationEndpointsPassTheSameBudgetSemantics"); + RunTest(TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint, "TestQueueCompletionIsExactlyOnceAndIncludesCompletionsEndpoint"); + RunTest(TestQueueCompletionReleasesImmediatelyOrDelaysQueuedHandoff, "TestQueueCompletionReleasesImmediatelyOrDelaysQueuedHandoff"); + RunTest(TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable, "TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable"); +#else + RunTest(TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib, "TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib"); + RunTest(TestFeatureOffRejectsRaiTagWithoutIncludingCorelibHeaders, "TestFeatureOffRejectsRaiTagWithoutIncludingCorelibHeaders"); +#endif + std::cout << "test_phi4_frontend: PASS\n"; +} diff --git a/src/test/phi4_rai/test_phi4_gguf.cpp b/src/test/phi4_rai/test_phi4_gguf.cpp new file mode 100644 index 000000000..9f15f4e09 --- /dev/null +++ b/src/test/phi4_rai/test_phi4_gguf.cpp @@ -0,0 +1,580 @@ +#include "gguf_fixture.hpp" +#include "fake_corelib.hpp" +#include "rai/corelib_api.hpp" +#include "models/phi4/rai/phi4_rai_constants.hpp" +#include "models/phi4/rai/phi4_rai_gguf.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +using flm::phi4::Phi4GgufPackage; +using gguf_fixture::Builder; +using gguf_fixture::Mutation; + +std::shared_ptr Open(Builder builder, + gguf_fixture::TempFile& file, + std::string_view label) { + file = builder.Write(label); + return Phi4GgufPackage::Open(file.path); +} + +std::string OpenFailure(Builder builder, Mutation mutation, + std::string_view label) { + auto file = builder.Apply(mutation).Write(label); + return RequireThrows([&] { Phi4GgufPackage::Open(file.path); }); +} + +void RequireMismatch(std::string_view error, std::string_view field, + std::string_view actual, std::string_view expected) { + RequireContains(error, field); + RequireContains(error, "actual " + std::string(actual)); + RequireContains(error, "expected " + std::string(expected)); +} + +std::string ShapeText(const std::vector& shape) { + std::string result = "["; + for (std::size_t index = 0; index < shape.size(); ++index) { + if (index != 0) result += ','; + result += std::to_string(shape[index]); + } + return result + ']'; +} + +struct TensorRole { + std::string name; + std::vector shape; + std::uint32_t type; +}; + +std::vector RequiredTensorRoles() { + std::vector roles = { + {"token_embd.weight", {200064, 3072}, gguf_fixture::kQ8_0}, + {"output_norm.weight", {3072}, gguf_fixture::kF32}}; + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto prefix = "blk." + std::to_string(layer); + roles.push_back({prefix + ".attn_norm.weight", {3072}, gguf_fixture::kF32}); + roles.push_back({prefix + ".ffn_norm.weight", {3072}, gguf_fixture::kF32}); + roles.push_back({prefix + ".attn_qkv.weight", {5120, 3072}, gguf_fixture::kQ8_0}); + roles.push_back({prefix + ".attn_output.weight", {3072, 3072}, gguf_fixture::kQ8_0}); + roles.push_back({prefix + ".ffn_up.weight", {16384, 3072}, gguf_fixture::kQ8_0}); + roles.push_back({prefix + ".ffn_down.weight", {3072, 8192}, gguf_fixture::kQ8_0}); + } + return roles; +} + +Builder SplitFixture() { + Builder builder; + builder.AddTensor("blk.0.attn_qkv.weight", {5120, 3072}, gguf_fixture::kQ8_0) + .AddTensor("blk.0.ffn_up.weight", {16384, 3072}, gguf_fixture::kQ8_0) + .AddTensor("f32", {48}, gguf_fixture::kF32); + return builder; +} + +void TestValidV3HeaderMetadataDirectoryAndAlignment() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "valid"); + const auto metadata = package->Metadata(); + TEST_REQUIRE(metadata.architecture == "phi3"); + TEST_REQUIRE(metadata.layer_count == 32); + TEST_REQUIRE(metadata.tokenizer_vocabulary_size == 200064); + TEST_REQUIRE(!metadata.add_bos_token); +} + +void TestOmittedAlignmentUsesGgufDefault32() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture().RemoveMetadata("general.alignment"), + file, "default-alignment"); + TEST_REQUIRE(package->RequireF32( + "f32", std::array{48}).values.size() == 48); +} + +void TestEveryMetadataScalarStringAndArrayEncodingCanBeSkippedSafely() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture().AddEverySkippableMetadataType(), file, + "metadata-types"); + TEST_REQUIRE(package->Metadata().hidden_size == 3072); +} + +void TestTruncatedHeaderMetadataStringArrayAndTensorDirectoryFail() { + const auto truncated_header = std::filesystem::temp_directory_path() / "flm_phi4_short_header.gguf"; + { std::ofstream out(truncated_header, std::ios::binary | std::ios::trunc); out << "GG"; } + RequireContains(RequireThrows([&] { Phi4GgufPackage::Open(truncated_header); }), "header"); + std::error_code ignored; std::filesystem::remove(truncated_header, ignored); + RequireContains(OpenFailure(SplitFixture(), Mutation::TruncatedString, "truncated-string"), "string"); + RequireContains(OpenFailure(SplitFixture(), Mutation::TruncatedDirectory, "truncated-directory"), "tensor"); + + Builder array; + array.RemoveMetadata("tokenizer.ggml.tokens") + .AddMetadata("tokenizer.ggml.tokens", gguf_fixture::ArrayValue{ + 8, std::numeric_limits::max(), {}}) + .AddTensor("x", {32}, gguf_fixture::kQ8_0); + auto file = array.Write("truncated-array"); + RequireContains(RequireThrows([&] { Phi4GgufPackage::Open(file.path); }), "array"); +} + +void TestCountProductAlignmentAndOffsetOverflowFail() { + RequireContains(OpenFailure(SplitFixture(), Mutation::CountOverflow, "count-overflow"), "count"); + RequireContains(OpenFailure(SplitFixture(), Mutation::ProductOverflow, "product-overflow"), "overflow"); + RequireContains(OpenFailure(SplitFixture(), Mutation::OffsetOverflow, "offset-overflow"), "overflow"); +} + +void TestPresentMalformedAlignmentFails() { + RequireContains(OpenFailure(SplitFixture(), Mutation::ZeroAlignment, "zero-align"), "alignment"); + RequireContains(OpenFailure(SplitFixture(), Mutation::NonPowerOfTwoAlignment, "bad-align"), "alignment"); + + auto wrong_type = SplitFixture().SetMetadata( + "general.alignment", std::int32_t{-32}).Write("wrong-align-type"); + RequireMismatch(RequireThrows([&] { Phi4GgufPackage::Open(wrong_type.path); }), + "general.alignment", "INT32", "unsigned integer metadata"); +} + +void TestDuplicateTensorNamesFail() { + RequireContains(OpenFailure(SplitFixture(), Mutation::DuplicateName, "duplicate"), "duplicate"); +} + +void TestOutOfFileAndOverlappingTensorRangesFail() { + RequireContains(OpenFailure(SplitFixture(), Mutation::OutOfFileRange, "outside"), "range"); + RequireContains(OpenFailure(SplitFixture(), Mutation::OverlappingRanges, "overlap"), "overlap"); + RequireContains(OpenFailure(SplitFixture(), Mutation::PayloadLengthMismatch, "short-payload"), "range"); +} + +void TestUnsupportedUnskippableMetadataTypeFails() { + RequireContains(OpenFailure(SplitFixture(), Mutation::UnsupportedMetadataType, "unsupported"), "metadata type"); +} + +void TestRequireQ8AndRequireF32ReportNameActualAndExpected() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "requires"); + auto error = RequireThrows([&] { package->RequireQ8("f32", std::array{48}); }); + RequireMismatch(error, "f32", "F32", "Q8_0"); + error = RequireThrows([&] { package->RequireF32("f32", std::array{47}); }); + RequireMismatch(error, "f32", "[48]", "[47]"); +} + +void TestAttentionQkvReturnsThreeZeroCopyWholeRowViews() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "qkv"); + const auto fused = package->RequireQ8("blk.0.attn_qkv.weight", std::array{5120, 3072}); + const auto views = package->AttentionQkv(0); + const std::size_t row_bytes = 3072 / 32 * 34; + TEST_REQUIRE(views.count == 3); + TEST_REQUIRE(views.values[0].bytes.data() == fused.bytes.data()); + TEST_REQUIRE(views.values[1].bytes.data() == fused.bytes.data() + 3072 * row_bytes); + TEST_REQUIRE(views.values[2].bytes.data() == fused.bytes.data() + 4096 * row_bytes); + TEST_REQUIRE(views.values[0].logical_shape == std::vector({3072, 3072})); + TEST_REQUIRE(views.values[1].logical_shape == std::vector({1024, 3072})); + TEST_REQUIRE(views.values[2].logical_shape == std::vector({1024, 3072})); +} + +void TestGateUpReturnsTwoZeroCopyWholeRowViews() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "gate-up"); + const auto fused = package->RequireQ8("blk.0.ffn_up.weight", std::array{16384, 3072}); + const auto views = package->GateUp(0); + const std::size_t row_bytes = 3072 / 32 * 34; + TEST_REQUIRE(views.count == 2); + TEST_REQUIRE(views.values[0].bytes.data() == fused.bytes.data()); + TEST_REQUIRE(views.values[1].bytes.data() == fused.bytes.data() + 8192 * row_bytes); + TEST_REQUIRE(views.values[0].logical_shape == std::vector({8192, 3072})); + TEST_REQUIRE(views.values[1].logical_shape == std::vector({8192, 3072})); +} + +void TestSplitRejectsNonIntegralQ8RowBoundary() { + Builder builder; + builder.AddTensor("blk.0.attn_qkv.weight", {5120, 3073}, gguf_fixture::kQ8_0); + auto file = builder.Write("bad-row"); + auto package = Phi4GgufPackage::Open(file.path); + RequireMismatch(RequireThrows([&] { package->AttentionQkv(0); }), + "blk.0.attn_qkv.weight", "3073", + "Q8_0 whole-row width divisible by 32"); +} + +void TestViewsPointIntoTheReadOnlyMapping() { + gguf_fixture::TempFile file; + auto package = Open(SplitFixture(), file, "mapping"); + const auto first = package->RequireF32("f32", std::array{48}); + const auto second = package->RequireF32("f32", std::array{48}); + TEST_REQUIRE(first.values.data() == second.values.data()); + TEST_REQUIRE(first.values.size() == 48); + + auto misaligned_file = Builder().AddTensor("misaligned-f32", {48}, gguf_fixture::kF32) + .Apply(Mutation::MisalignedF32).Write("misaligned-f32"); + auto misaligned = Phi4GgufPackage::Open(misaligned_file.path); + const auto error = RequireThrows([&] { + misaligned->RequireF32("misaligned-f32", std::array{48}); + }); + RequireMismatch(error, "misaligned-f32", "address", "alignment 4"); +} + +struct ContractFixture { + gguf_fixture::TempFile file; + std::shared_ptr package; + ContractFixture() { + package = Open(Builder().AddFullContractTensors(), file, "contract"); + } +}; + +void TestAcceptsExactPhi3Phi4Contract() { + ContractFixture fixture; + fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), + gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); +} + +void TestRejectsWrongArchitectureAndEveryDimension() { + struct Case { + std::string field; + gguf_fixture::MetadataValue value; + std::string actual; + std::string expected; + }; + const std::vector cases = { + {"general.architecture", std::string("llama"), "llama", "phi3"}, + {"phi3.block_count", std::uint32_t{31}, "31", "32"}, + {"phi3.context_length", std::uint32_t{131071}, "131071", "131072"}, + {"phi3.embedding_length", std::uint32_t{3071}, "3071", "3072"}, + {"phi3.feed_forward_length", std::uint32_t{8191}, "8191", "8192"}, + {"phi3.attention.head_count", std::uint32_t{23}, "23", "24"}, + {"phi3.attention.head_count_kv", std::uint32_t{7}, "7", "8"}, + {"phi3.rope.dimension_count", std::uint32_t{95}, "95", "96"}, + {"tokenizer.ggml.tokens", + gguf_fixture::ArrayValue{0, 200063, std::vector(200063)}, + "200063", "200064"}}; + for (const auto& test_case : cases) { + auto file = Builder().SetMetadata(test_case.field, test_case.value) + .AddFullContractTensors().Write("wrong-field"); + auto package = Phi4GgufPackage::Open(file.path); + const auto error = RequireThrows([&] { package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); + RequireMismatch(error, test_case.field, test_case.actual, test_case.expected); + } +} + +void TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole() { + ContractFixture valid; + for (std::size_t layer = 0; layer < 32; ++layer) { + const auto prefix = "blk." + std::to_string(layer); + valid.package->RequireF32(prefix + ".attn_norm.weight", std::array{3072}); + valid.package->RequireF32(prefix + ".ffn_norm.weight", std::array{3072}); + valid.package->RequireQ8(prefix + ".attn_qkv.weight", std::array{5120, 3072}); + valid.package->RequireQ8(prefix + ".attn_output.weight", std::array{3072, 3072}); + valid.package->RequireQ8(prefix + ".ffn_up.weight", std::array{16384, 3072}); + valid.package->RequireQ8(prefix + ".ffn_down.weight", std::array{3072, 8192}); + } + const nlohmann::json unused; + for (const auto& role : RequiredTensorRoles()) { + auto missing_file = Builder().AddFullContractTensors().RemoveTensor(role.name).Write("missing-role"); + auto missing = Phi4GgufPackage::Open(missing_file.path); + RequireMismatch(RequireThrows([&] { missing->ValidatePhi4Contract(unused, unused, unused); }), + role.name, "missing", "present tensor"); + + const auto wrong_type = role.type == gguf_fixture::kQ8_0 + ? gguf_fixture::kF32 : gguf_fixture::kQ8_0; + auto type_file = Builder().AddFullContractTensors() + .MutateTensor(role.name, wrong_type, role.shape).Write("wrong-type"); + auto type_package = Phi4GgufPackage::Open(type_file.path); + RequireMismatch(RequireThrows([&] { type_package->ValidatePhi4Contract(unused, unused, unused); }), + role.name, wrong_type == gguf_fixture::kF32 ? "F32" : "Q8_0", + role.type == gguf_fixture::kF32 ? "F32" : "Q8_0"); + + auto wrong_shape = role.shape; + --wrong_shape.front(); + auto shape_file = Builder().AddFullContractTensors() + .MutateTensor(role.name, role.type, wrong_shape).Write("wrong-shape"); + auto shape_package = Phi4GgufPackage::Open(shape_file.path); + const auto shape_error = RequireThrows([&] { + shape_package->ValidatePhi4Contract(unused, unused, unused); + }); + RequireMismatch(shape_error, role.name, ShapeText(wrong_shape), ShapeText(role.shape)); + + auto length_file = Builder().AddFullContractTensors() + .TruncateTensorPayload(role.name).Write("wrong-length"); + const auto length_error = RequireThrows([&] { Phi4GgufPackage::Open(length_file.path); }); + RequireMismatch(length_error, role.name + " range", "out-of-file range", + "range within mapped file"); + } +} + +void TestRejectsMixedQuantizationAndOutputWeightPresence() { + auto mixed_file = Builder().AddFullContractTensors().MutateTensor("token_embd.weight", gguf_fixture::kF32, {200064,3072}).Write("mixed"); + auto mixed = Phi4GgufPackage::Open(mixed_file.path); + RequireMismatch(RequireThrows([&] { mixed->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "token_embd.weight", "F32", "Q8_0"); + auto output_file = Builder().AddFullContractTensors().AddTensor("output.weight", {200064,3072}, gguf_fixture::kQ8_0).Write("output-weight"); + auto output = Phi4GgufPackage::Open(output_file.path); + const auto error = RequireThrows([&] { output->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); + RequireMismatch(error, "output.weight", "present", "absent (tied token_embd.weight)"); +} + +void TestRequiresTiedQ8TokenEmbeddingAsLmHead() { + auto file = Builder().AddFullContractTensors().RemoveTensor("token_embd.weight").Write("untied"); + auto package = Phi4GgufPackage::Open(file.path); + RequireMismatch(RequireThrows([&] { package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "token_embd.weight", "missing", "present tensor"); +} + +void TestRequiresOriginal4096WindowAndValidatesLongRopeFactors() { + auto wrong_file = Builder().SetMetadata("phi3.rope.scaling.original_context_length", std::uint32_t{8192}).AddFullContractTensors().Write("long-window"); + auto wrong = Phi4GgufPackage::Open(wrong_file.path); + RequireMismatch(RequireThrows([&] { wrong->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "phi3.rope.scaling.original_context_length", "8192", "4096"); + + auto valid_file = Builder().AddFullContractTensors().AddTensor( + "rope_factors_long.weight", {48}, gguf_fixture::kF32).Write("long-rope"); + auto valid = Phi4GgufPackage::Open(valid_file.path); + valid->ValidatePhi4Contract(gguf_fixture::ValidConfig(), + gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); + + auto malformed_file = Builder().AddFullContractTensors().AddTensor( + "rope_factors_long.weight", {47}, gguf_fixture::kF32).Write("bad-long-rope"); + auto malformed = Phi4GgufPackage::Open(malformed_file.path); + RequireMismatch(RequireThrows([&] { malformed->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "rope_factors_long.weight", "[47]", "[48]"); +} + +void TestValidatesOptionalShortRopeFactorsAsF32Length48() { + auto absent_file = Builder().AddFullContractTensors(false).Write("no-short-rope"); + auto absent = Phi4GgufPackage::Open(absent_file.path); + absent->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); + auto wrong_file = Builder().AddFullContractTensors(false).AddTensor("rope_factors_short.weight", {47}, gguf_fixture::kF32).Write("wrong-short-rope"); + auto wrong = Phi4GgufPackage::Open(wrong_file.path); + RequireMismatch(RequireThrows([&] { wrong->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + "rope_factors_short.weight", "[47]", "[48]"); +} + +void TestRejectsNonFiniteOrNonPositiveRopeValues() { + for (const auto& field : {"phi3.rope.freq_base", "phi3.rope.scaling.attn_factor"}) { + for (const float value : {0.0f, -1.0f, std::numeric_limits::infinity(), std::numeric_limits::quiet_NaN()}) { + auto file = Builder().SetMetadata(field, value).AddFullContractTensors().Write("bad-rope-value"); + auto package = Phi4GgufPackage::Open(file.path); + RequireMismatch(RequireThrows([&] { package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }), + field, std::to_string(static_cast(value)), + "finite positive value"); + } + } +} + +void TestOmittedHeadDimUsesHiddenSizeDividedByAttentionHeads() { + ContractFixture fixture; + auto config = gguf_fixture::ValidConfig(); + config.erase("head_dim"); + fixture.package->ValidatePhi4Contract( + config, gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); +} + +void TestRejectsConfigDisagreement() { + ContractFixture fixture; + struct Case { + std::string field; + nlohmann::json value; + std::string actual; + std::string expected; + }; + const std::vector cases = { + {"model_type", "other", "other", "phi3"}, + {"num_hidden_layers", 31, "31", "32"}, + {"hidden_size", 3071, "3071", "3072"}, + {"intermediate_size", 8191, "8191", "8192"}, + {"num_attention_heads", 23, "23", "24"}, + {"num_key_value_heads", 7, "7", "8"}, + {"head_dim", 127, "127", "128"}, + {"vocab_size", 200063, "200063", "200064"}, + {"rms_norm_eps", 2.0e-5, "2e-05", "0.000010"}, + {"original_max_position_embeddings", 4095, "4095", "4096"}, + {"hidden_size", 3072.0, "3072.0", "integer 3072"}, + {"hidden_size", std::uint64_t{4294970368ULL}, "4294970368", "3072"}, + {"eos_token_id", 199999.0, "199999.0", "integer 199999"}}; + for (const auto& test_case : cases) { + auto config = gguf_fixture::ValidConfig(); + config[test_case.field] = test_case.value; + const auto error = RequireThrows([&] { fixture.package->ValidatePhi4Contract( + config, gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); }); + RequireMismatch(error, test_case.field, test_case.actual, test_case.expected); + } +} + +void TestDerivesStopSetFromGgufConfigAndTokenizerIds() { + ContractFixture fixture; + fixture.package->ValidatePhi4Contract(gguf_fixture::ValidConfig(), + gguf_fixture::ValidTokenizer(), gguf_fixture::ValidTokenizerConfig()); + + auto gguf_file = Builder().SetMetadata("tokenizer.ggml.eos_token_id", std::uint32_t{1}) + .AddFullContractTensors().Write("wrong-gguf-eos"); + auto gguf = Phi4GgufPackage::Open(gguf_file.path); + RequireMismatch(RequireThrows([&] { gguf->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.ggml.eos_token_id", "1", "200020"); + + auto config = gguf_fixture::ValidConfig(); + config["eos_token_id"] = 1; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + config, gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "eos_token_id", "1", "199999"); + + for (const auto& [token, expected] : std::array{ + std::pair{"<|end|>", 200020}, + std::pair{"<|endoftext|>", 199999}}) { + auto tokenizer = gguf_fixture::ValidTokenizer(); + tokenizer["model"]["vocab"][token] = 1; + for (auto& added : tokenizer["added_tokens"]) + if (added["content"] == token) added["id"] = 1; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), tokenizer, + gguf_fixture::ValidTokenizerConfig()); }), + token, "1", std::to_string(expected)); + } +} + +void TestAcceptsPinnedDynamicRoleChatTemplate() { + ContractFixture fixture; + auto tokenizer_config = gguf_fixture::ValidTokenizerConfig(); + tokenizer_config["chat_template"] = + "{% for message in messages %}{{ '<|' + message['role'] + '|>' + " + "message['content'] + '<|end|>' }}{% endfor %}" + "{% if add_generation_prompt %}{{ '<|assistant|>' }}{% endif %}"; + fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), tokenizer_config); +} + +void TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement() { + ContractFixture fixture; + + auto tokenizer = gguf_fixture::ValidTokenizer(); + tokenizer["model"]["vocab"].erase("t0"); + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), tokenizer, + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.json distinct vocabulary ID count", "200028", "200029"); + + for (const auto& [invalid_id, actual, expected] : std::array{ + std::tuple{-1, "-1", "0..200063"}, + std::tuple{200064, "200064", "0..200063"}, + std::tuple{ + std::numeric_limits::max(), "18446744073709551615", "0..200063"}, + std::tuple{0.0, "0.0", "integer in 0..200063"}}) { + tokenizer = gguf_fixture::ValidTokenizer(); + tokenizer["model"]["vocab"]["t0"] = invalid_id; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), tokenizer, + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.json token ID", actual, expected); + } + + tokenizer = gguf_fixture::ValidTokenizer(); + auto& added = tokenizer["added_tokens"]; + const auto highest = std::find_if(added.begin(), added.end(), [](const auto& item) { + return item.at("id") == 200028; + }); + TEST_REQUIRE(highest != added.end()); + added.erase(highest); + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), tokenizer, + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.json maximum vocabulary ID", "200027", "200028"); + + auto bos_file = Builder().SetMetadata("tokenizer.ggml.add_bos_token", true) + .AddFullContractTensors().Write("wrong-gguf-bos"); + auto bos = Phi4GgufPackage::Open(bos_file.path); + RequireMismatch(RequireThrows([&] { bos->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "tokenizer.ggml.add_bos_token", "true", "false"); + + auto tokenizer_config = gguf_fixture::ValidTokenizerConfig(); + tokenizer_config["add_bos_token"] = true; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + tokenizer_config); }), + "add_bos_token", "true", "false"); + tokenizer_config = gguf_fixture::ValidTokenizerConfig(); + tokenizer_config["chat_template"] = "<|user|><|assistant|>"; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + tokenizer_config); }), + "<|end|>", "missing from chat_template", "present in chat_template"); +} + +void TestRejectsFiniteWrongRmsValue() { + auto file = Builder().SetMetadata("phi3.attention.layer_norm_rms_epsilon", 2.0e-5f) + .AddFullContractTensors().Write("wrong-rms"); + auto package = Phi4GgufPackage::Open(file.path); + const auto error = RequireThrows([&] { package->ValidatePhi4Contract( + gguf_fixture::ValidConfig(), gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }); + RequireMismatch(error, "phi3.attention.layer_norm_rms_epsilon", + "0.000020", "0.000010"); +} + +void TestValidationCreatesNoCorelibObjects() { + fake_corelib::Reset(); + auto api = flm::corelib::CorelibApi::ResolveForTest(fake_corelib::Resolver()); + fake_corelib::GetState().call_counts.clear(); + + ContractFixture fixture; + auto config = gguf_fixture::ValidConfig(); + config["hidden_size"] = 1; + RequireMismatch(RequireThrows([&] { fixture.package->ValidatePhi4Contract( + config, gguf_fixture::ValidTokenizer(), + gguf_fixture::ValidTokenizerConfig()); }), + "hidden_size", "1", "3072"); + + for (const auto name : {"ryzenai_corelib_create_stream", + "ryzenai_corelib_create_device_tensor", + "ryzenai_corelib_create_tensor_window", + "ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized", + "ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized"}) + TEST_REQUIRE(fake_corelib::GetState().call_counts[name] == 0); + TEST_REQUIRE(fake_corelib::GetState().live_objects == 0); + (void)api; +} + +} // namespace + +int main() { +#define RUN(name) RunTest(name, #name) + RUN(TestValidV3HeaderMetadataDirectoryAndAlignment); + RUN(TestOmittedAlignmentUsesGgufDefault32); + RUN(TestEveryMetadataScalarStringAndArrayEncodingCanBeSkippedSafely); + RUN(TestTruncatedHeaderMetadataStringArrayAndTensorDirectoryFail); + RUN(TestCountProductAlignmentAndOffsetOverflowFail); + RUN(TestPresentMalformedAlignmentFails); + RUN(TestDuplicateTensorNamesFail); + RUN(TestOutOfFileAndOverlappingTensorRangesFail); + RUN(TestUnsupportedUnskippableMetadataTypeFails); + RUN(TestRequireQ8AndRequireF32ReportNameActualAndExpected); + RUN(TestAttentionQkvReturnsThreeZeroCopyWholeRowViews); + RUN(TestGateUpReturnsTwoZeroCopyWholeRowViews); + RUN(TestSplitRejectsNonIntegralQ8RowBoundary); + RUN(TestViewsPointIntoTheReadOnlyMapping); + RUN(TestAcceptsExactPhi3Phi4Contract); + RUN(TestRejectsWrongArchitectureAndEveryDimension); + RUN(TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole); + RUN(TestRejectsMixedQuantizationAndOutputWeightPresence); + RUN(TestRequiresTiedQ8TokenEmbeddingAsLmHead); + RUN(TestRequiresOriginal4096WindowAndValidatesLongRopeFactors); + RUN(TestValidatesOptionalShortRopeFactorsAsF32Length48); + RUN(TestRejectsNonFiniteOrNonPositiveRopeValues); + RUN(TestOmittedHeadDimUsesHiddenSizeDividedByAttentionHeads); + RUN(TestRejectsConfigDisagreement); + RUN(TestRejectsFiniteWrongRmsValue); + RUN(TestDerivesStopSetFromGgufConfigAndTokenizerIds); + RUN(TestAcceptsPinnedDynamicRoleChatTemplate); + RUN(TestRejectsTokenizerVocabularyEosBosAndMarkerDisagreement); + RUN(TestValidationCreatesNoCorelibObjects); +#undef RUN + return 0; +} diff --git a/src/test/phi4_rai/test_phi4_host.cpp b/src/test/phi4_rai/test_phi4_host.cpp new file mode 100644 index 000000000..862875686 --- /dev/null +++ b/src/test/phi4_rai/test_phi4_host.cpp @@ -0,0 +1,185 @@ +#include "models/phi4/rai/phi4_rai_host.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define NOMINMAX +#include + +namespace { +using namespace flm::phi4; + +void PutHalf(std::vector& bytes, std::size_t offset, std::uint16_t bits) { + bytes[offset] = static_cast(bits & 0xff); + bytes[offset + 1] = static_cast(bits >> 8); +} + +TensorView ThreeRows() { + static std::vector bytes(3 * 34, std::byte{0x7f}); + std::fill(bytes.begin(), bytes.end(), std::byte{0x7f}); + for (std::size_t row = 0; row < 3; ++row) { + const std::size_t base = row * 34; + PutHalf(bytes, base, row == 0 ? 0x3800 : row == 1 ? 0x3c00 : 0x4000); + for (std::size_t column = 0; column < 32; ++column) { + const auto value = static_cast(row == 1 ? -static_cast(column) : + static_cast(row + column)); + bytes[base + 2 + column] = static_cast(value); + } + } + return {"token_embd.weight", bytes, {3, 32}, 8}; +} + +GgufPhi4Metadata Metadata(double attention = 1.0) { + return {"phi3", 32, 3072, 8192, 24, 8, 131072, 96, 10000.0, + attention, 4096, 200064, false}; +} + +void TestLazyEmbeddingDecodesOnlyRequestedRows() { + constexpr std::size_t width = 65536; + constexpr std::size_t row_bytes = width / 32 * 34; // 17 Windows pages. + auto* mapping = static_cast(VirtualAlloc( + nullptr, 3 * row_bytes, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE)); + TEST_REQUIRE(mapping != nullptr); + for (std::size_t block = 0; block < width / 32; ++block) { + const std::uint16_t scale = 0x3c00; + std::memcpy(mapping + row_bytes + block * 34, &scale, sizeof(scale)); + std::fill_n(mapping + row_bytes + block * 34 + 2, 32, std::byte{0xff}); + } + DWORD old_protection{}; + TEST_REQUIRE(VirtualProtect(mapping, row_bytes, PAGE_NOACCESS, &old_protection)); + TEST_REQUIRE(VirtualProtect(mapping + 2 * row_bytes, row_bytes, + PAGE_NOACCESS, &old_protection)); + const TensorView embedding{"token_embd.weight", {mapping, 3 * row_bytes}, + {3, static_cast(width)}, 8}; + const std::array ids{1}; + const auto decoded = DecodeEmbeddingRowsQ8(embedding, ids); + TEST_REQUIRE(decoded.size() == width); + TEST_REQUIRE(decoded.front() == -1.0f && decoded.back() == -1.0f); + VirtualFree(mapping, 0, MEM_RELEASE); +} + +void TestLazyEmbeddingPreservesRequestOrderAndDuplicates() { + const auto embedding = ThreeRows(); + const std::array ids{2, 0, 2}; + const auto decoded = DecodeEmbeddingRowsQ8(embedding, ids); + TEST_REQUIRE(decoded.size() == 96); + TEST_REQUIRE(decoded[0] == 4.0f); + TEST_REQUIRE(decoded[32] == 0.0f); + TEST_REQUIRE(decoded[64] == 4.0f); + TEST_REQUIRE(decoded[95] == 66.0f); +} + +void TestLazyEmbeddingRejectsNegativeAndOutOfRangeIds() { + const auto embedding = ThreeRows(); + std::array negative{-1}; + std::array too_large{3}; + RequireContains(RequireThrows([&] { DecodeEmbeddingRowsQ8(embedding, negative); }), + "token id"); + RequireContains(RequireThrows([&] { DecodeEmbeddingRowsQ8(embedding, too_large); }), + "token id"); + auto malformed = embedding; + malformed.bytes = malformed.bytes.first(malformed.bytes.size() - 1); + std::array valid{0}; + RequireContains(RequireThrows([&] { DecodeEmbeddingRowsQ8(malformed, valid); }), + "Q8_0"); +} + +void TestHostRmsNormUsesDoubleAccumulationAndMatchesReferenceBits() { + const std::array input{ + std::bit_cast(0xBE8BBBACu), + std::bit_cast(0xBCCC9DE0u), + std::bit_cast(0xBFED682Fu), + std::bit_cast(0xC2CD01EDu)}; + const std::array scale{1.0f, 1.0f, 1.0f, 1.0f}; + std::array output{}; + HostRmsNorm(input, scale, 1, 4, 1.0e-5f, output); + constexpr std::array expected{ + 0xBBAE75DBu, 0xB9FF7820u, 0xBD143451u, 0xBFFFF50Bu}; + for (std::size_t i = 0; i < output.size(); ++i) + TEST_REQUIRE(std::bit_cast(output[i]) == expected[i]); +} + +void TestHostRmsNormRejectsZeroAndShapeErrors() { + std::array input{1.0f, 2.0f}; + std::array scale{1.0f, 1.0f}; + std::array output{}; + RequireContains(RequireThrows([&] { HostRmsNorm(input, scale, 0, 2, 1.0e-5f, output); }), "positive"); + RequireContains(RequireThrows([&] { HostRmsNorm(input, scale, 1, 0, 1.0e-5f, output); }), "positive"); + RequireContains(RequireThrows([&] { HostRmsNorm(input, std::span(scale).first(1), 1, 2, 1.0e-5f, output); }), "shape"); + RequireContains(RequireThrows([&] { HostRmsNorm(input, scale, 1, 2, -1.0f, output); }), "epsilon"); +} + +void TestHostRmsNormMatchesPr706Bf16BoundaryReference() { + constexpr std::size_t width = 3072; + std::vector input(width, 0.03125f); + input[0] = 1024.0f; + std::vector scale(width, 1.0f); + std::vector output(width); + HostRmsNorm(input, scale, 1, width, 1.0e-5f, output); + const auto bf16 = ConvertF32ToBf16(output); + TEST_REQUIRE(std::bit_cast(output[0]) == 0x425DB3C3u); + TEST_REQUIRE(std::bit_cast(output[1]) == 0x3ADDB3C3u); + TEST_REQUIRE(bf16[0] == 0x425e); + TEST_REQUIRE(bf16[1] == 0x3ade); +} + +void TestF32ToBf16UsesRoundToNearestEven() { + const std::array values{ + std::bit_cast(std::uint32_t{0x3f808000}), + std::bit_cast(std::uint32_t{0x3f818000}), + -2.5f, + std::numeric_limits::infinity()}; + const auto result = ConvertF32ToBf16(values); + TEST_REQUIRE(result == std::vector({0x3f80, 0x3f82, 0xc020, 0x7f80})); +} + +void TestRopeTablesUseFloat64IntermediatesAndFloat32Outputs() { + const auto tables = BuildShortRopeTables(Metadata(), std::nullopt); + constexpr std::size_t i = 47; + constexpr std::size_t p = 4095; + const double inv = 1.0 / std::pow(10000.0, (2.0 * i) / 96.0); + const float expected = static_cast(std::cos(p * inv)); + TEST_REQUIRE(tables.cosine[p * 48 + i] == expected); +} + +void TestRopeTablesApplyShortFactorsAndAttentionFactor() { + std::array factors{}; + factors.fill(2.0f); + FloatTensorView factor_view{"rope_factors_short.weight", factors, {48}}; + const auto tables = BuildShortRopeTables(Metadata(1.5), factor_view); + const double inv = 1.0 / (std::pow(10000.0, 2.0 / 96.0) * 2.0); + TEST_REQUIRE(tables.cosine[48 + 1] == static_cast(std::cos(inv) * 1.5)); + TEST_REQUIRE(tables.sine[48 + 1] == static_cast(std::sin(inv) * 1.5)); +} + +void TestRopeTablesHaveShape4096By48() { + const auto tables = BuildShortRopeTables(Metadata(), std::nullopt); + TEST_REQUIRE(tables.cosine.size() == 4096 * 48); + TEST_REQUIRE(tables.sine.size() == 4096 * 48); + TEST_REQUIRE(tables.cosine[0] == 1.0f); + TEST_REQUIRE(tables.sine[0] == 0.0f); +} +} // namespace + +int main() { +#define RUN_TEST(name) RunTest(&name, #name) + RUN_TEST(TestLazyEmbeddingDecodesOnlyRequestedRows); + RUN_TEST(TestLazyEmbeddingPreservesRequestOrderAndDuplicates); + RUN_TEST(TestLazyEmbeddingRejectsNegativeAndOutOfRangeIds); + RUN_TEST(TestHostRmsNormUsesDoubleAccumulationAndMatchesReferenceBits); + RUN_TEST(TestHostRmsNormRejectsZeroAndShapeErrors); + RUN_TEST(TestHostRmsNormMatchesPr706Bf16BoundaryReference); + RUN_TEST(TestF32ToBf16UsesRoundToNearestEven); + RUN_TEST(TestRopeTablesUseFloat64IntermediatesAndFloat32Outputs); + RUN_TEST(TestRopeTablesApplyShortFactorsAndAttentionFactor); + RUN_TEST(TestRopeTablesHaveShape4096By48); +#undef RUN_TEST +} diff --git a/src/test/phi4_rai/test_phi4_shape_plan.cpp b/src/test/phi4_rai/test_phi4_shape_plan.cpp new file mode 100644 index 000000000..75b5ac343 --- /dev/null +++ b/src/test/phi4_rai/test_phi4_shape_plan.cpp @@ -0,0 +1,123 @@ +#include "models/phi4/rai/phi4_rai_shape_plan.hpp" +#include "models/phi4/rai/phi4_rai_constants.hpp" +#include "fake_corelib.hpp" +#include "test_support.hpp" + +#include +#include +#include +#include + +namespace { +using flm::corelib::CorelibApi; +using flm::phi4::Phi4ShapePlan; + +std::shared_ptr Api() { + return CorelibApi::ResolveForTest(fake_corelib::Resolver()); +} + +/// \brief a stream to build a plan against +/// \note Since corelib 0.5.0 every padding helper takes the stream, +/// because the PDI pair it was opened with selects the kernel set. +/// The fake hands back an opaque object; nothing dereferences it. +ryzenai_corelib_stream_ptr Stream(const std::shared_ptr& api) { + void* raw = nullptr; + api->Check(api->functions().create_stream(flm::phi4::kPrefillPdi, + flm::phi4::kTokenPdi, &raw), + "ryzenai_corelib_create_stream"); + return raw; +} + +void TestShapePlanQueriesOnlyExecutionBucketsAndMapsEveryRow() { + fake_corelib::Reset(); + const auto api = Api(); + const auto plan = Phi4ShapePlan::Build(api, Stream(api)); + const auto& state = fake_corelib::GetState(); + constexpr std::array buckets{ + 1, 64, 128, 256, 512, 1024, 2048, 4096}; + TEST_REQUIRE(state.matmul_pad_calls.size() == 3 * buckets.size() + 1); + TEST_REQUIRE(state.rows_pad_calls.size() == buckets.size()); + TEST_REQUIRE(state.mha_pad_calls.size() == buckets.size()); + for (std::size_t index = 0; index < buckets.size(); ++index) { + TEST_REQUIRE(state.matmul_pad_calls[index * 3].m == buckets[index]); + TEST_REQUIRE(state.matmul_pad_calls[index * 3].group_size == 64); + TEST_REQUIRE(state.rows_pad_calls[index].m == buckets[index]); + TEST_REQUIRE(state.mha_pad_calls[index].m == buckets[index]); + } + TEST_REQUIRE(plan.ForRows(2).query_rows == 64); + TEST_REQUIRE(plan.ForRows(65).query_rows == 128); + TEST_REQUIRE(plan.ForRows(257).query_rows == 512); + TEST_REQUIRE(plan.ForRows(4095).query_rows == 4096); +} + +void TestShapePlanUsesExactQKvOutputSsmlpRmsAndLmHeadDimensions() { + fake_corelib::Reset(); + const auto api = Api(); + const auto plan = Phi4ShapePlan::Build(api, Stream(api)); + const auto& state = fake_corelib::GetState(); + const auto& q = state.matmul_pad_calls[0]; + const auto& kv = state.matmul_pad_calls[1]; + const auto& output = state.matmul_pad_calls[2]; + TEST_REQUIRE(q.k == 3072 && q.n == 3072); + TEST_REQUIRE(kv.k == 3072 && kv.n == 1024); + TEST_REQUIRE(output.k == 3072 && output.n == 3072); + TEST_REQUIRE(state.rows_pad_calls[0].helper == "ssmlp"); + TEST_REQUIRE(state.rows_pad_calls[0].k == 3072); + TEST_REQUIRE(state.rows_pad_calls[0].n == 8192); + const auto& lm = state.matmul_pad_calls.back(); + TEST_REQUIRE(lm.m == 1 && lm.k == 3072 && lm.n == 200064 && lm.group_size == 64); + TEST_REQUIRE(plan.lm_head_desc().k == 3072); + TEST_REQUIRE(plan.lm_head_desc().n == 200064); +} + +void TestShapePlanBuildsFlatMhaDescriptor24_8_128_4096_96() { + fake_corelib::Reset(); + const auto api = Api(); + const auto plan = Phi4ShapePlan::Build(api, Stream(api)); + const auto& desc = plan.attention_desc(); + TEST_REQUIRE(desc.num_heads == 24); + TEST_REQUIRE(desc.kv_num_heads == 8); + TEST_REQUIRE(desc.head_size == 128); + TEST_REQUIRE(desc.max_seq == 4096); + TEST_REQUIRE(desc.rope_dim == 96); + TEST_REQUIRE(fake_corelib::GetState().mha_pad_calls.front().desc.rope_dim == 96); +} + +void TestShapePlanRejectsPaddedKOrNChanges() { + fake_corelib::Reset(); + fake_corelib::GetState().matmul_k_delta = 1; + RequireContains(RequireThrows([&] { const auto a = Api(); Phi4ShapePlan::Build(a, Stream(a)); }), "padded K/N"); + fake_corelib::Reset(); + fake_corelib::GetState().matmul_n_delta = 1; + RequireContains(RequireThrows([&] { const auto a = Api(); Phi4ShapePlan::Build(a, Stream(a)); }), "padded K/N"); +} + +void TestShapePlanRejectsRowsOutsideCachedRange() { + fake_corelib::Reset(); + const auto api = Api(); + const auto plan = Phi4ShapePlan::Build(api, Stream(api)); + RequireContains(RequireThrows([&] { plan.ForRows(0); }), "1..4096"); + RequireContains(RequireThrows([&] { plan.ForRows(4097); }), "1..4096"); +} + +void TestShapePlanFailureNamesHelperAndLogicalShape() { + fake_corelib::Reset(); + auto api = Api(); + fake_corelib::GetState().statuses["ryzenai_corelib_ssmlp_bf16_pad_rows"] = + ryzenai_corelib_status_unsupported; + const auto error = RequireThrows([&] { Phi4ShapePlan::Build(api, Stream(api)); }); + RequireContains(error, "ryzenai_corelib_ssmlp_bf16_pad_rows"); + RequireContains(error, "[1,3072,8192]"); +} +} // namespace + +int main() { +#define RUN_TEST(name) RunTest(&name, #name) + RUN_TEST(TestShapePlanQueriesOnlyExecutionBucketsAndMapsEveryRow); + RUN_TEST(TestShapePlanUsesExactQKvOutputSsmlpRmsAndLmHeadDimensions); + RUN_TEST(TestShapePlanBuildsFlatMhaDescriptor24_8_128_4096_96); + RUN_TEST(TestShapePlanRejectsPaddedKOrNChanges); + RUN_TEST(TestShapePlanRejectsRowsOutsideCachedRange); + RUN_TEST(TestShapePlanFailureNamesHelperAndLogicalShape); +#undef RUN_TEST +} diff --git a/src/test/phi4_rai/test_real_corelib.cpp b/src/test/phi4_rai/test_real_corelib.cpp new file mode 100644 index 000000000..feb841a0a --- /dev/null +++ b/src/test/phi4_rai/test_real_corelib.cpp @@ -0,0 +1,38 @@ +#include "rai/corelib_api.hpp" +#include "rai/corelib_runtime.hpp" +#include "test_support.hpp" + +#include +#include +#include + +int main() { + const char* configured = std::getenv("FLM_RAI_CORELIB_PATH"); + if (configured == nullptr || *configured == '\0') { + std::cout << "SKIP: FLM_RAI_CORELIB_PATH is unset\n"; + return 77; + } + + try { + const auto api = flm::corelib::CorelibApi::Load( + flm::corelib::CorelibApi::ResolveLibraryPath( + std::filesystem::current_path())); + const auto version = api->runtime_version(); + // The pin this build was compiled against, not a literal: the + // point is that the DLL on this box is the one FastFlowLM was + // built for, whichever that is. + TEST_REQUIRE(version.major == RYZENAI_CORELIB_VERSION_MAJOR && + version.minor == RYZENAI_CORELIB_VERSION_MINOR && + version.patch == RYZENAI_CORELIB_VERSION_PATCH); +#define FLM_ASSERT_CORELIB_SYMBOL(member, symbol) TEST_REQUIRE(api->functions().member != nullptr); + FLM_CORELIB_FUNCTIONS(FLM_ASSERT_CORELIB_SYMBOL) +#undef FLM_ASSERT_CORELIB_SYMBOL + auto runtime = flm::corelib::CorelibRuntime::CreateForTest(api); + runtime.reset(); + flm::corelib::CorelibRuntime::ShutdownProcess(); + } catch (const std::exception& error) { + std::cerr << error.what() << '\n'; + return 1; + } + return 0; +} diff --git a/src/test/phi4_rai/test_support.hpp b/src/test/phi4_rai/test_support.hpp new file mode 100644 index 000000000..1fc1efd8f --- /dev/null +++ b/src/test/phi4_rai/test_support.hpp @@ -0,0 +1,61 @@ +#pragma once + +#include +#include +#include +#include +#include +#include + +#define TEST_REQUIRE(condition) \ + do { \ + if (!(condition)) { \ + throw std::runtime_error(std::string("requirement failed: ") + \ + #condition); \ + } \ + } while (false) + +inline void RequireContains(std::string_view text, std::string_view expected) { + if (text.find(expected) == std::string_view::npos) { + throw std::runtime_error("expected '" + std::string(text) + + "' to contain '" + std::string(expected) + "'"); + } +} + +template +std::string RequireThrows(Callable&& callable) { + try { + callable(); + } catch (const Exception& error) { + return error.what(); + } + throw std::runtime_error("expected exception was not thrown"); +} + +/// \brief how many tests have failed so far in this binary +/// \note Exiting on the first failure hides every later one, so a suite with +/// three broken tests looks like a suite with one and each fix uncovers +/// the next. Keep going and fail the process at the end instead. +inline int& FailureCount() { + static int failures = 0; + return failures; +} + +inline void RunTest(void (*test)(), const char* name) { + try { + test(); + std::cout << "PASS " << name << '\n'; + } catch (const std::exception& error) { + std::cerr << "FAIL " << name << ": " << error.what() << '\n'; + // Every main here ends with an unconditional "PASS" and return 0, so + // the process exit code has to be forced from here. atexit runs after + // main returns, which is late enough to have counted every test. + if (++FailureCount() == 1) { + std::atexit([] { + std::cerr << FailureCount() << " test(s) FAILED\n"; + std::cerr.flush(); + std::_Exit(1); + }); + } + } +} From 47cdd53da34b5049672333846ec50aa83f04fd8e Mon Sep 17 00:00:00 2001 From: alfxu_amdeng Date: Wed, 23 Sep 2026 14:00:04 -0700 Subject: [PATCH 02/17] feat(rai): build the ryzenai-corelib backend on Linux FLM_ENABLE_RAI was rejected at configure time on anything but Windows, and the code behind the gate had grown three Win32-only spots to match. Nothing about the backend is actually Windows-specific: ryzenai-corelib's own C++ already carries POSIX branches throughout, and DynamicDispatch builds on Linux today. Only FastFlowLM's side was holding it shut. - CMakeLists: drop the FATAL_ERROR, search lib64/ beside lib/ for the corelib import library, and make the Boost lookup Windows-only. Boost is needed only because MSVC reports __cplusplus as 199711L without /Zc:__cplusplus, which sends xrt/detail/any.h down its boost::any branch; GCC and Clang report C++20 and never take it. - CMakeLists: flm_rai named only the singular XRT_INCLUDE_DIR, which the pkg-config path never sets, so the target would have had no XRT headers at all on Linux. It now picks the same variable the flm target does, and links CMAKE_DL_LIBS for the dlopen path that still compiles in a static build. - phi4_rai_gguf.cpp: mmap the GGUF through open/fstat/mmap where the Win32 file-mapping calls were the only option. Read-only and MAP_PRIVATE, with O_CLOEXEC so a mapping that lives as long as the engine cannot leak into a spawned process. - corelib_api.cpp: Load() dlopens instead of throwing "requires Windows". RTLD_LOCAL keeps corelib's symbols out of the global namespace, since this process links XRT directly and a corelib built against a different one must not satisfy our calls; RTLD_NOW surfaces a missing symbol next to the path that was loaded rather than at first dispatch. The three inline ".dll" spellings collapse into two constants, so FLM_RAI_CORELIB_PATH diagnostics cannot describe a suffix other than the one required. - Add a linux-rai preset mirroring windows-rai, and update the docs that asserted the backend was Windows-only. Verified by configuring and building on Linux against corelib 0.5.0 headers: flm_rai compiles all eight translation units, and the flm target compiles clean with main.cpp's FLM_ENABLE_RAI path wired to it. No NPU hardware here, so nothing is run; src/test/phi4_rai stays Windows-gated and untouched. Co-Authored-By: Claude Opus 5 --- docs/docs/models/phi.md | 22 ++++-- src/CMakeLists.txt | 47 ++++++++++--- src/CMakePresets.json | 20 ++++++ src/common/models/README.md | 12 +++- src/common/models/phi4/rai/phi4_rai_gguf.cpp | 53 ++++++++++++++ src/common/rai/corelib_api.cpp | 72 +++++++++++++++----- src/include/rai/corelib_api.hpp | 5 +- 7 files changed, 195 insertions(+), 36 deletions(-) diff --git a/docs/docs/models/phi.md b/docs/docs/models/phi.md index e838af7a3..c6dee6ac6 100644 --- a/docs/docs/models/phi.md +++ b/docs/docs/models/phi.md @@ -31,7 +31,7 @@ flm run phi4-mini-it:4b - **Source format:** GGUF, read directly. No ONNX model, no tensor manifest, and no converted or packed weight file is produced or shipped. - **Quantization:** GGML `Q8_0` in the file, requantized to **group 64** while the weights are packed for the device, through corelib's explicit `*_create_gguf_requantized` entry points. This is a **lossy** second quantization step and it is not reversible; output will differ from the Q8_0 source. - **Usable generation window:** 4095 tokens — the rendered prompt plus the requested output together, so the largest admissible prompt is 4094. An over-capacity request is rejected with HTTP 400 *before* any work is submitted to the device. Note this is far below the model's 128k context; see below for why. -- **Availability:** Windows only, and this is a **developer build**. The rai runtime is not packaged by the MSI or Inno installer; you build against corelib yourself. +- **Availability:** Windows and Linux, and this is a **developer build**. The rai runtime is not packaged by the MSI, Inno or snap installer; you build against corelib yourself. On rai this tag pulls from two pinned repositories, because the GGUF publisher does not ship the tokenizer files FastFlowLM's tokenizer frontend consumes: @@ -52,17 +52,29 @@ From `FastFlowLM/src`, in a Visual Studio developer command prompt: ```powershell $env:RYZENAI_CORELIB_INCLUDE_DIR = 'C:/path/to/ryzenai-corelib/install/include' -cmake --preset windows-rai # sets FLM_ENABLE_RAI=ON, builds into src/build-rai +$env:RYZENAI_CORELIB_LIBRARY = 'C:/path/to/ryzenai-corelib/install/lib/ryzenai_corelib.lib' +cmake --preset windows-rai -DFLM_ENABLE_RAI=ON # builds into src/build-rai cmake --build --preset windows-rai ``` -The `windows-rai` preset reads `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY` from the environment, so set both before configuring. The configure step also locates a Boost include directory, and hard-errors if the option is enabled on a non-Windows host. Everything else — XRT, FFmpeg, curl, FFTW — is the ordinary FastFlowLM dependency set; the rai option does not relax any of it. +or, on Linux: + +```shell +export RYZENAI_CORELIB_INCLUDE_DIR=/path/to/ryzenai-corelib/install/include +export RYZENAI_CORELIB_LIBRARY=/path/to/ryzenai-corelib/install/lib/libryzenai_corelib.so +cmake --preset linux-rai -DFLM_ENABLE_RAI=ON # builds into src/build-rai +cmake --build --preset linux-rai +``` + +Both presets read `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY` from the environment, so set both before configuring. They ship with `FLM_ENABLE_RAI` off so that the preset still configures on a machine without corelib, which is why the option is passed on the command line above. The configure step additionally locates a Boost include directory on Windows only — XRT's `xrt/detail/any.h` reaches for `boost::any` when `__cplusplus` reads below 201703L, which MSVC does unless it is handed `/Zc:__cplusplus`; GCC and Clang report C++20 honestly, so nothing there needs Boost. Everything else — XRT, FFmpeg, curl, FFTW — is the ordinary FastFlowLM dependency set; the rai option does not relax any of it. + +The `src/test/phi4_rai` suite is still Windows-only and is not configured on Linux; a Linux build is a compile-and-link path, not a tested one. ### Pointing FastFlowLM at the runtime -A rai build (`-DFLM_ENABLE_RAI=ON`) **links corelib in**, because the NPU device the whole process shares comes from corelib's `ryzenai::corelib::GetDevice()` rather than from a device FastFlowLM opens itself. Point the build at the library with `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY`. `FLM_RAI_CORELIB_PATH` selects a DLL only in the older dynamically loading configuration; in a statically linked rai build it is ignored, and `flm` says so if it is set. +A rai build (`-DFLM_ENABLE_RAI=ON`) **links corelib in**, because the NPU device the whole process shares comes from corelib's `ryzenai::corelib::GetDevice()` rather than from a device FastFlowLM opens itself. Point the build at the library with `RYZENAI_CORELIB_INCLUDE_DIR` and `RYZENAI_CORELIB_LIBRARY`. `FLM_RAI_CORELIB_PATH` selects a shared library (`.dll` on Windows, `.so` elsewhere) only in the older dynamically loading configuration; in a statically linked rai build it is ignored, and `flm` says so if it is set. -The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.3.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. Corelib's own dependency directory must be reachable on `PATH`. +The corelib ABI is still pre-1.0, so FastFlowLM requires an **exact `0.3.0`** match on major, minor and patch. The version is queried before any other entry point, so a mismatched runtime reports a version error rather than a missing symbol. Corelib's own dependency directory must be reachable on `PATH` (Windows) or `LD_LIBRARY_PATH` (Linux). ```powershell flm pull phi4-mini-it:4b diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 254fd0564..9c3daedf5 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -55,19 +55,29 @@ option(FLM_ENABLE_RAI "Reach kernels through ryzenai-corelib instead of FastFlowLM's own flow" OFF) if(FLM_ENABLE_RAI) - if(NOT WIN32) - message(FATAL_ERROR "FLM_ENABLE_RAI currently requires Windows") - endif() find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h REQUIRED) # The main binary needs ryzenai::corelib::GetDevice(), a C++ entry point the # C ABI does not expose, so corelib is linked in rather than dlopened. + # A conda/pip prefix keeps its import libs beside the headers; a CMake + # install prefix puts them in lib/ or lib64/ depending on the distribution, + # so both are offered rather than assuming either. find_library(RYZENAI_CORELIB_LIBRARY NAMES ryzenai_corelib corelib HINTS "${RYZENAI_CORELIB_INCLUDE_DIR}/../lib" + "${RYZENAI_CORELIB_INCLUDE_DIR}/../lib64" "$ENV{RYZENAI_CORELIB_LIB_DIR}" REQUIRED) - find_path(FLM_CORELIB_BOOST_INCLUDE_DIR NAMES boost/any.hpp - HINTS "$ENV{CONDA_PREFIX}/Library/include" - "$ENV{USERPROFILE}/anaconda3/Library/include" - "C:/dev/boost_1_88_0" REQUIRED) + # XRT's xrt/detail/any.h selects boost::any over std::any whenever + # __cplusplus reads below 201703L. MSVC reports 199711L regardless of /std + # unless it is handed /Zc:__cplusplus, so a Windows build has to supply + # Boost headers that nothing else on the rai path wants. GCC and Clang + # report C++20 honestly, so that branch is never taken there and there is + # nothing to find -- which is why this is a Windows-only lookup rather than + # a dependency of the rai backend. + if(WIN32) + find_path(FLM_CORELIB_BOOST_INCLUDE_DIR NAMES boost/any.hpp + HINTS "$ENV{CONDA_PREFIX}/Library/include" + "$ENV{USERPROFILE}/anaconda3/Library/include" + "C:/dev/boost_1_88_0" REQUIRED) + endif() endif() if(FLM_USE_HRX) @@ -298,14 +308,29 @@ if(FLM_ENABLE_RAI) include("${CMAKE_SOURCE_DIR}/common/rai/rai_sources.cmake") add_library(flm_rai STATIC ${FLM_RAI_SOURCES}) target_include_directories(flm_rai PUBLIC - "${CMAKE_SOURCE_DIR}/include" "${RYZENAI_CORELIB_INCLUDE_DIR}" - "${XRT_INCLUDE_DIR}" "${FLM_CORELIB_BOOST_INCLUDE_DIR}") + "${CMAKE_SOURCE_DIR}/include" "${RYZENAI_CORELIB_INCLUDE_DIR}") + # XRT headers come from whichever discovery above succeeded: pkg-config sets + # XRT_INCLUDE_DIRS, the Windows and /opt/xilinx fallbacks set the singular + # XRT_INCLUDE_DIR. Same choice the flm target makes below -- naming only the + # singular one here left flm_rai with no XRT headers at all on Linux. + if(NOT WIN32 AND XRT_FOUND) + target_include_directories(flm_rai PUBLIC ${XRT_INCLUDE_DIRS}) + else() + target_include_directories(flm_rai PUBLIC ${XRT_INCLUDE_DIR}) + endif() + # Only Windows looks this one up; see the find_path above for why. + if(FLM_CORELIB_BOOST_INCLUDE_DIR) + target_include_directories(flm_rai PUBLIC "${FLM_CORELIB_BOOST_INCLUDE_DIR}") + endif() # RYZENAI_CORELIB_STATIC drops the vendor header's dllimport decoration; # FLM_CORELIB_LINK_STATIC is what tells our own code the symbols are linked - # in and must be bound directly instead of through LoadLibrary. + # in and must be bound directly instead of through LoadLibrary/dlopen. target_compile_definitions(flm_rai PUBLIC FLM_ENABLE_RAI=1 RYZENAI_CORELIB_STATIC=1 FLM_CORELIB_LINK_STATIC=1) - target_link_libraries(flm_rai PUBLIC "${RYZENAI_CORELIB_LIBRARY}") + # CMAKE_DL_LIBS is -ldl on Linux and empty on Windows: corelib_api.cpp still + # compiles its dlopen path in a statically linked build, so the symbols have + # to resolve even though nothing calls them here. + target_link_libraries(flm_rai PUBLIC "${RYZENAI_CORELIB_LIBRARY}" ${CMAKE_DL_LIBS}) target_link_libraries(flm PRIVATE flm_rai) endif() diff --git a/src/CMakePresets.json b/src/CMakePresets.json index 54fde039f..38c7ff228 100644 --- a/src/CMakePresets.json +++ b/src/CMakePresets.json @@ -31,6 +31,18 @@ "FLM_PORTABLE_BUILD": "ON" } }, + { + "name": "linux-rai", + "displayName": "Linux RAI", + "description": "Linux build with statically linked ryzenai-corelib support", + "inherits": "linux-default", + "binaryDir": "${sourceDir}/build-rai", + "cacheVariables": { + "FLM_ENABLE_RAI": "OFF", + "RYZENAI_CORELIB_INCLUDE_DIR": "$env{RYZENAI_CORELIB_INCLUDE_DIR}", + "RYZENAI_CORELIB_LIBRARY": "$env{RYZENAI_CORELIB_LIBRARY}" + } + }, { "name": "linux-snap", "displayName": "Linux Snap Build", @@ -98,6 +110,14 @@ "configuration": "Release", "jobs": 4 }, + { + "name": "linux-rai", + "displayName": "Linux RAI Build", + "description": "Build the optional ryzenai-corelib runtime in Release", + "configurePreset": "linux-rai", + "configuration": "Release", + "jobs": 4 + }, { "name": "windows-rai", "displayName": "Windows RAI Build", diff --git a/src/common/models/README.md b/src/common/models/README.md index eed016c71..e24929bfa 100644 --- a/src/common/models/README.md +++ b/src/common/models/README.md @@ -21,19 +21,25 @@ This is a contributor document. For *using* a backend once it exists — `--back | | | |---|---| -| **Platform** | Windows only. `FLM_ENABLE_RAI` is rejected at configure time elsewhere ([`CMakeLists.txt:54`](../../CMakeLists.txt#L54)). | +| **Platform** | Windows and Linux. Both configure and build; only Windows has been run on hardware, and the `src/test/phi4_rai` suite is still Windows-only. | | **Hardware** | An aie_next NPU. There is no simulator; a wrong shape shows up as garbage output, not an error. | | **corelib headers** | Exactly **0.5.0**. [`corelib_api.hpp`](../../include/rai/corelib_api.hpp) `#error`s on any other version — deliberately, because the C ABI has changed shape between patch releases. | | **Weights** | A GGUF the vendor kernels can requantize. Phi-4 uses Q8_0; the corelib entry points are `*_create_gguf_requantized`. | Configure with: -```powershell +```shell cmake -B build -S src -DFLM_ENABLE_RAI=ON # RYZENAI_CORELIB_INCLUDE_DIR / RYZENAI_CORELIB_LIB_DIR are found automatically -# when they are on the default paths; otherwise pass them. +# when they are on the default paths; otherwise pass them. The library lookup +# searches lib/ and lib64/ beside the headers as well as RYZENAI_CORELIB_LIB_DIR. ``` +Boost is looked up only on Windows: XRT's `xrt/detail/any.h` falls back to +`boost::any` whenever `__cplusplus` reads below 201703L, which MSVC does +unless handed `/Zc:__cplusplus`. GCC and Clang report C++20 honestly, so that +branch is never taken and there is nothing to find. + That builds the `flm_rai` static library and links it into `flm`. The library carries `FLM_ENABLE_RAI=1` as a **PUBLIC** compile definition, so everything that links it sees the `#if` guards flip. diff --git a/src/common/models/phi4/rai/phi4_rai_gguf.cpp b/src/common/models/phi4/rai/phi4_rai_gguf.cpp index 278a52e0a..bf8061a93 100644 --- a/src/common/models/phi4/rai/phi4_rai_gguf.cpp +++ b/src/common/models/phi4/rai/phi4_rai_gguf.cpp @@ -3,8 +3,18 @@ #include "models/phi4/rai/phi4_rai_constants.hpp" #include "utils/file_access.hpp" +#ifdef _WIN32 +#ifndef NOMINMAX #define NOMINMAX +#endif #include +#else +#include +#include +#include +#include +#include +#endif #include #include @@ -266,8 +276,15 @@ struct Phi4GgufPackage::Impl { }; std::filesystem::path path; + // The mapping handles differ by platform; everything below them -- `data`, + // `size` and the parsed tables -- does not, so only these two lines and the + // destructor and Open() below are conditional. +#ifdef _WIN32 HANDLE file = INVALID_HANDLE_VALUE; HANDLE mapping = nullptr; +#else + int file = -1; +#endif const std::byte* data = nullptr; std::uint64_t size = 0; std::map> tensors; @@ -275,9 +292,17 @@ struct Phi4GgufPackage::Impl { std::map> metadata_types; ~Impl() { +#ifdef _WIN32 if (data) UnmapViewOfFile(data); if (mapping) CloseHandle(mapping); if (file != INVALID_HANDLE_VALUE) CloseHandle(file); +#else + // munmap takes the size the mapping was made with, so it has to run + // before anything clears `size`; nothing here does. + if (data) + ::munmap(const_cast(data), static_cast(size)); + if (file >= 0) ::close(file); +#endif } std::span bytes() const { @@ -335,6 +360,7 @@ std::shared_ptr Phi4GgufPackage::Open( auto impl = std::make_unique(); impl->path = gguf_path; flm::file_access::ObserveOpen(gguf_path); +#ifdef _WIN32 impl->file = CreateFileW(gguf_path.c_str(), GENERIC_READ, FILE_SHARE_READ, nullptr, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, nullptr); if (impl->file == INVALID_HANDLE_VALUE) @@ -354,6 +380,33 @@ std::shared_ptr Phi4GgufPackage::Open( if (!impl->data) throw std::runtime_error("GGUF mapping: actual MapViewOfFile failure " + std::to_string(GetLastError()) + ", expected FILE_MAP_READ view"); +#else + // The POSIX half of the same three steps: open, size, map read-only. errno + // stands in for GetLastError() -- the numbers differ but the shape of the + // message does not, so a failure reads the same on either platform. + // O_CLOEXEC because a GGUF stays mapped for the life of the engine and must + // not leak into anything the process spawns. + impl->file = ::open(gguf_path.c_str(), O_RDONLY | O_CLOEXEC); + if (impl->file < 0) + throw std::runtime_error("GGUF file: actual open failure " + + std::to_string(errno) + ", expected readable file"); + struct ::stat status {}; + if (::fstat(impl->file, &status) != 0 || status.st_size <= 0 || + static_cast(status.st_size) > + std::numeric_limits::max()) + Fail("GGUF file size", + std::to_string(static_cast(status.st_size)), + "positive mappable size"); + impl->size = static_cast(status.st_size); + // MAP_PRIVATE, not MAP_SHARED: the mapping is read-only and nothing writes + // back, and a private mapping does not pin the pages against another writer. + void* view = ::mmap(nullptr, static_cast(impl->size), PROT_READ, + MAP_PRIVATE, impl->file, 0); + if (view == MAP_FAILED) + throw std::runtime_error("GGUF mapping: actual mmap failure " + + std::to_string(errno) + ", expected PROT_READ view"); + impl->data = static_cast(view); +#endif const auto file = impl->bytes(); Cursor cursor(file); diff --git a/src/common/rai/corelib_api.cpp b/src/common/rai/corelib_api.cpp index 1b5733a0d..a438eb048 100644 --- a/src/common/rai/corelib_api.cpp +++ b/src/common/rai/corelib_api.cpp @@ -12,6 +12,8 @@ #define NOMINMAX #endif #include +#else +#include #endif namespace flm::corelib { @@ -45,13 +47,33 @@ std::string ErrorText(std::string_view call, return result; } -bool HasDllExtension(const std::filesystem::path& path) { +/// \brief what a shared library is called on this platform +/// \note Only the spelling changes between platforms, so the extension and the +/// file name are named once here rather than at each of the three places +/// that used to write ".dll" inline. FLM_RAI_CORELIB_PATH is validated +/// against the same constants, so the diagnostic cannot describe a +/// different suffix from the one actually required. +constexpr std::string_view kSharedLibraryExtension = +#ifdef _WIN32 + ".dll"; +#else + ".so"; +#endif + +constexpr std::string_view kCorelibLibraryName = +#ifdef _WIN32 + "ryzenai_corelib.dll"; +#else + "libryzenai_corelib.so"; +#endif + +bool HasSharedLibraryExtension(const std::filesystem::path& path) { std::string extension = path.extension().string(); std::transform(extension.begin(), extension.end(), extension.begin(), [](unsigned char value) { return static_cast(std::tolower(value)); }); - return extension == ".dll"; + return extension == kSharedLibraryExtension; } } // namespace @@ -106,18 +128,15 @@ std::shared_ptr CorelibApi::ResolveForTest( std::move(resolver), std::move(loaded_library_path))); } -std::shared_ptr CorelibApi::Load(const std::filesystem::path& dll) { -#ifndef _WIN32 - (void)dll; - throw std::runtime_error("ryzenai-corelib loading currently requires Windows"); -#else - const std::filesystem::path absolute_dll = std::filesystem::absolute(dll); +std::shared_ptr CorelibApi::Load(const std::filesystem::path& library) { + const std::filesystem::path absolute_library = std::filesystem::absolute(library); +#ifdef _WIN32 HMODULE raw_module = LoadLibraryExW( - absolute_dll.c_str(), nullptr, + absolute_library.c_str(), nullptr, LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_DEFAULT_DIRS); if (!raw_module) { - throw std::runtime_error("failed to load corelib DLL '" + - absolute_dll.string() + "' (Windows error " + + throw std::runtime_error("failed to load corelib library '" + + absolute_library.string() + "' (Windows error " + std::to_string(GetLastError()) + ")"); } auto module = std::shared_ptr(raw_module, [](void* handle) { @@ -128,8 +147,27 @@ std::shared_ptr CorelibApi::Load(const std::filesystem::path& dll) { return reinterpret_cast( GetProcAddress(static_cast(module.get()), terminated.c_str())); }; - return ResolveForTest(std::move(resolver), absolute_dll); +#else + // RTLD_LOCAL so corelib's symbols do not join the global namespace: this + // process also links XRT directly, and a corelib built against a different + // one must not be able to satisfy our XRT calls. RTLD_NOW because a missing + // symbol has to surface here, next to the path that was loaded, rather than + // at the first dispatch. + void* raw_module = ::dlopen(absolute_library.c_str(), RTLD_NOW | RTLD_LOCAL); + if (!raw_module) { + const char* reason = ::dlerror(); + throw std::runtime_error("failed to load corelib library '" + + absolute_library.string() + "' (" + + (reason ? reason : "unknown dlopen failure") + ")"); + } + auto module = std::shared_ptr( + raw_module, [](void* handle) { ::dlclose(handle); }); + Resolver resolver = [module](std::string_view name) -> void* { + const std::string terminated(name); + return ::dlsym(module.get(), terminated.c_str()); + }; #endif + return ResolveForTest(std::move(resolver), absolute_library); } #if defined(FLM_CORELIB_LINK_STATIC) @@ -163,16 +201,18 @@ std::filesystem::path CorelibApi::ResolveLibraryPath( const std::filesystem::path path(configured); if (!path.is_absolute()) { throw std::runtime_error( - "FLM_RAI_CORELIB_PATH must be an absolute .dll path"); + "FLM_RAI_CORELIB_PATH must be an absolute " + + std::string(kSharedLibraryExtension) + " path"); } - if (!path.has_filename() || !HasDllExtension(path)) { + if (!path.has_filename() || !HasSharedLibraryExtension(path)) { throw std::runtime_error( - "FLM_RAI_CORELIB_PATH must name an absolute .dll file"); + "FLM_RAI_CORELIB_PATH must name an absolute " + + std::string(kSharedLibraryExtension) + " file"); } return path; } return std::filesystem::absolute(executable_dir / "rai" / - "ryzenai_corelib.dll"); + kCorelibLibraryName); } const CorelibFunctions& CorelibApi::functions() const noexcept { return functions_; } diff --git a/src/include/rai/corelib_api.hpp b/src/include/rai/corelib_api.hpp index f6355bf44..16f5c6edd 100644 --- a/src/include/rai/corelib_api.hpp +++ b/src/include/rai/corelib_api.hpp @@ -82,7 +82,10 @@ struct CorelibFunctions { class CorelibApi final { public: using Resolver = std::function; - static std::shared_ptr Load(const std::filesystem::path& dll); + /// \brief load corelib from a shared library at run time + /// \note LoadLibraryEx on Windows, dlopen elsewhere. Unused by a build that + /// links corelib in (see LoadStatic), but still compiled there. + static std::shared_ptr Load(const std::filesystem::path& library); #if defined(FLM_CORELIB_LINK_STATIC) /// \brief bind to the corelib linked into this binary /// \return the API bound to the linked symbols From 14e4aa357d84a388b02db667902450d40ba9b453 Mon Sep 17 00:00:00 2001 From: alfxu_amdeng Date: Wed, 23 Sep 2026 17:12:01 -0700 Subject: [PATCH 03/17] build(rai): make home_install.sh produce a working corelib install Building the rai backend on Linux worked; installing one did not. Seven separate faults, none of which report themselves as what they are. The build could not find corelib. Its headers live outside the repository, so a rai configure needs to be told where they are once; CMakeLists.txt now takes RYZENAI_CORELIB_ROOT and looks in src/lib for the library, which is where an out-of-tree corelib build is staged. The linux-rai preset described a rai build without enabling one, so linux-rai-on sets FLM_ENABLE_RAI and stops the description from being the only thing that says "rai". home_install.sh then wedged itself. A configure that fails still writes CMakeCache.txt but never reaches build.ninja, and the guard read the cache alone as "already configured" -- so every later run skipped the configure it needed and died with "ninja: error: loading 'build.ninja'", an error no amount of re-running clears. Key the decision on the generator file, which only a successful configure writes, and use --fresh so a retry does not inherit what it is replacing. A cache configured the other way round was reused just as silently, installing a stock build for --rai and the reverse; compare the cached FLM_ENABLE_RAI against what was asked for, unless the preset was named by hand, since a custom preset's intent is not ours to infer. BUILD_DIR defaulted to the repository root while every preset's binaryDir is src/build, so this script and `cmake --build --preset` were building two separate trees of the same project. A rai prefix carries corelib and the DynamicDispatch core it links against on top of the usual engines, which is roughly 550M -- too much for a home directory. --rai now installs to /scratch/$USER/flm_exe_rai, falling back to ~/flm_exe_rai where there is no scratch, with FLM_PREFIX still overriding either. Finally the staging copied corelib plus whatever ldd reported, which is the wrong closure. DynamicDispatch was configured with DD_MDS_IN_BINS_DLL=ON, so the entire AIE4 kernel set lives in libdyn_bins.so, and nothing declares it DT_NEEDED: Transaction::load_large_txn_ops_dll() dladdr()s itself and dlopen()s the file from beside libdyn_dispatch_core.so. ldd cannot see it, and when it is missing that loader returns silently, leaving an empty shape table and a first matmul that dies with "Shape list size: 0" -- which reads like an unsupported model rather than a missing file. Remember the directory libdyn_dispatch_core.so came from and stage libdyn_bins.so out of it. Co-Authored-By: Claude Opus 5 --- src/CMakeLists.txt | 20 ++++- src/CMakePresets.json | 17 ++++ src/home_install.sh | 184 ++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 212 insertions(+), 9 deletions(-) diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 9c3daedf5..798383c16 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -55,14 +55,30 @@ option(FLM_ENABLE_RAI "Reach kernels through ryzenai-corelib instead of FastFlowLM's own flow" OFF) if(FLM_ENABLE_RAI) - find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h REQUIRED) + # The corelib prefix lives outside this repository, so its headers have to + # be found rather than assumed. The hints mirror the library lookup below: + # a header tree vendored into include/ wins, then a prefix named by + # RYZENAI_CORELIB_ROOT, and failing both the standard CMake search still + # honours CMAKE_PREFIX_PATH. Without the hints every configure has to be + # handed a prefix on the command line, which is the only reason the rai + # build ever needed an environment variable. + find_path(RYZENAI_CORELIB_INCLUDE_DIR NAMES ryzenai/corelib.h + HINTS "${CMAKE_SOURCE_DIR}/include" + "${RYZENAI_CORELIB_ROOT}/include" + "$ENV{RYZENAI_CORELIB_ROOT}/include" REQUIRED) # The main binary needs ryzenai::corelib::GetDevice(), a C++ entry point the # C ABI does not expose, so corelib is linked in rather than dlopened. # A conda/pip prefix keeps its import libs beside the headers; a CMake # install prefix puts them in lib/ or lib64/ depending on the distribution, # so both are offered rather than assuming either. + # lib/ is where this repository already keeps the binaries it vendors, so a + # corelib dropped in beside them is found without anyone passing a path. + # It is searched first: a build that has deliberately been given a corelib + # should use that one rather than whatever an unrelated prefix happens to + # hold. find_library(RYZENAI_CORELIB_LIBRARY NAMES ryzenai_corelib corelib - HINTS "${RYZENAI_CORELIB_INCLUDE_DIR}/../lib" + HINTS "${CMAKE_SOURCE_DIR}/lib" + "${RYZENAI_CORELIB_INCLUDE_DIR}/../lib" "${RYZENAI_CORELIB_INCLUDE_DIR}/../lib64" "$ENV{RYZENAI_CORELIB_LIB_DIR}" REQUIRED) # XRT's xrt/detail/any.h selects boost::any over std::any whenever diff --git a/src/CMakePresets.json b/src/CMakePresets.json index 38c7ff228..f7b09c793 100644 --- a/src/CMakePresets.json +++ b/src/CMakePresets.json @@ -43,6 +43,15 @@ "RYZENAI_CORELIB_LIBRARY": "$env{RYZENAI_CORELIB_LIBRARY}" } }, + { + "name": "linux-rai-on", + "displayName": "Linux RAI (enabled)", + "description": "Linux build with the ryzenai-corelib kernel backend turned on. Builds into build/ like every other preset. Point it at a corelib prefix once with -DRYZENAI_CORELIB_ROOT=, or drop the headers into include/ and libryzenai_corelib.so into lib/.", + "inherits": "linux-default", + "cacheVariables": { + "FLM_ENABLE_RAI": "ON" + } + }, { "name": "linux-snap", "displayName": "Linux Snap Build", @@ -118,6 +127,14 @@ "configuration": "Release", "jobs": 4 }, + { + "name": "linux-rai-on", + "displayName": "Linux RAI Build (enabled)", + "description": "Build the ryzenai-corelib kernel backend in Release", + "configurePreset": "linux-rai-on", + "configuration": "Release", + "jobs": 4 + }, { "name": "windows-rai", "displayName": "Windows RAI Build", diff --git a/src/home_install.sh b/src/home_install.sh index 86e9bf50e..b955094d8 100755 --- a/src/home_install.sh +++ b/src/home_install.sh @@ -8,23 +8,49 @@ # Usage: # ./home_install.sh # build (if needed) + install to ~/flm_exe # ./home_install.sh --no-build # install an existing build/ tree only +# ./home_install.sh --rai # ryzenai-corelib backend, to scratch # FLM_PREFIX=/path ./home_install.sh # +# --rai installs to /scratch/$USER/flm_exe_rai rather than ~/flm_exe: a rai +# build and a stock one would otherwise overwrite each other, and the corelib +# and DynamicDispatch libraries staged into a rai prefix are far too large for a +# home directory. Falls back to ~/flm_exe_rai with no scratch space, and +# FLM_PREFIX overrides either default. +# +# --rai selects the linux-rai-on preset. Point that build at a corelib prefix +# with RYZENAI_CORELIB_ROOT=/path (once -- CMake caches it), or drop the headers +# into include/ and libryzenai_corelib.so into lib/. An explicit PRESET= in the +# environment overrides --rai. +# set -euo pipefail # ---- configuration --------------------------------------------------------- SRC_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" REPO_DIR="$(cd "$SRC_DIR/.." && pwd)" -FLM_PREFIX="${FLM_PREFIX:-$HOME/flm_exe}" -BUILD_DIR="${BUILD_DIR:-$REPO_DIR/build}" -PRESET="${PRESET:-linux-default}" +# The default prefix depends on --rai, which is not parsed yet, so the decision +# is deferred to just after the argument loop. Only the environment value is +# captured here so that an explicit FLM_PREFIX keeps overriding both defaults. +FLM_PREFIX_FROM_ENV="${FLM_PREFIX:-}" +# Every preset in CMakePresets.json puts its binaryDir at ${sourceDir}/build, +# and sourceDir is this directory -- not the repository root. Defaulting to the +# root instead left `cmake --build --preset` and this script configuring two +# separate trees of the same project, doubling the build and making --no-build +# miss a tree that had just been built. A PRESET with a different binaryDir +# (linux-rai, linux-snap) needs BUILD_DIR set to match it. +BUILD_DIR="${BUILD_DIR:-$SRC_DIR/build}" +# An explicit PRESET in the environment always wins; --rai only changes the +# default, which is why the environment value is remembered separately. +PRESET_FROM_ENV="${PRESET:-}" +PRESET="${PRESET_FROM_ENV:-linux-default}" # Where XRT (the AMD NPU runtime) is installed. Override if non-standard. XRT_DIR="${XRT_DIR:-/opt/xilinx/xrt}" DO_BUILD=1 +WANT_RAI=0 for arg in "$@"; do case "$arg" in --no-build) DO_BUILD=0 ;; + --rai) WANT_RAI=1 ;; -h|--help) grep '^#' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' exit 0 ;; @@ -32,15 +58,89 @@ for arg in "$@"; do esac done +if [[ "$WANT_RAI" -eq 1 && -z "$PRESET_FROM_ENV" ]]; then + PRESET="linux-rai-on" +fi + +# A rai install and a stock one differ in the binary and in the libraries staged +# beside it, but occupy identical paths, so sharing a prefix means whichever ran +# last silently replaces the other. Giving rai its own default keeps both usable +# and lets the two env scripts be sourced independently. +# +# That default lives on scratch rather than under $HOME: a rai install also +# carries libryzenai_corelib.so and the DynamicDispatch core it links against, +# which together dwarf the rest of the tree and do not belong in a home +# directory. $HOME is the fallback for a machine with no scratch space. +RAI_PREFIX_DEFAULT="/scratch/$USER/flm_exe_rai" +[[ -d "/scratch/$USER" ]] || RAI_PREFIX_DEFAULT="$HOME/flm_exe_rai" +if [[ -n "$FLM_PREFIX_FROM_ENV" ]]; then + FLM_PREFIX="$FLM_PREFIX_FROM_ENV" +elif [[ "$WANT_RAI" -eq 1 ]]; then + FLM_PREFIX="$RAI_PREFIX_DEFAULT" +else + FLM_PREFIX="$HOME/flm_exe" +fi + echo "[home_install] repo: $REPO_DIR" echo "[home_install] prefix: $FLM_PREFIX" echo "[home_install] build: $BUILD_DIR" +echo "[home_install] preset: $PRESET" # ---- build (optional) ------------------------------------------------------ +# A configure that fails part way still writes CMakeCache.txt, but never gets +# as far as emitting build.ninja. Treating the cache alone as "already +# configured" therefore wedged the script permanently: every later run skipped +# the configure it needed and died inside the build step with +# "ninja: error: loading 'build.ninja'", which says nothing about the real +# fault. Key the decision on the generator file, which only exists once a +# configure has actually succeeded. +needs_configure() { + [[ ! -f "$BUILD_DIR/CMakeCache.txt" ]] && return 0 + # Ninja and Makefile cover the generators these presets select; anything + # else is left alone rather than guessed at. + [[ -f "$BUILD_DIR/build.ninja" || -f "$BUILD_DIR/Makefile" ]] || return 0 + # Reusing a cache configured the other way round would silently install a + # non-rai build for --rai (or the reverse). Only checked when the preset was + # not chosen by hand, since a custom preset's intent is not ours to infer. + if [[ -z "$PRESET_FROM_ENV" ]]; then + local cached=OFF + grep -q '^FLM_ENABLE_RAI:BOOL=ON' "$BUILD_DIR/CMakeCache.txt" && cached=ON + local wanted=OFF + [[ "$WANT_RAI" -eq 1 ]] && wanted=ON + [[ "$cached" == "$wanted" ]] || return 0 + fi + return 1 +} + if [[ "$DO_BUILD" -eq 1 ]]; then - if [[ ! -f "$BUILD_DIR/build.ninja" && ! -f "$BUILD_DIR/CMakeCache.txt" ]]; then + if needs_configure; then echo "[home_install] configuring (preset: $PRESET) ..." - cmake -S "$SRC_DIR" -B "$BUILD_DIR" --preset "$PRESET" + configure_args=() + if [[ -n "${RYZENAI_CORELIB_ROOT:-}" ]]; then + configure_args+=(-DRYZENAI_CORELIB_ROOT="$RYZENAI_CORELIB_ROOT") + fi + # --fresh discards whatever a previous failed or differently-configured + # run left behind. Without it CMake reloads those entries and the + # reconfigure inherits the state it is meant to replace. + fresh_args=() + [[ -f "$BUILD_DIR/CMakeCache.txt" ]] && fresh_args+=(--fresh) + if ! cmake -S "$SRC_DIR" -B "$BUILD_DIR" --preset "$PRESET" \ + ${fresh_args[@]+"${fresh_args[@]}"} \ + ${configure_args[@]+"${configure_args[@]}"}; then + echo >&2 + echo "[home_install] ERROR: configure failed." >&2 + if [[ "$WANT_RAI" -eq 1 || "$PRESET" == *rai* ]]; then + echo " A rai build needs the ryzenai-corelib headers. If the error" >&2 + echo " above is about RYZENAI_CORELIB_INCLUDE_DIR, point the build at" >&2 + echo " the corelib prefix:" >&2 + echo >&2 + echo " RYZENAI_CORELIB_ROOT=/path/to/ryzenai-corelib \\" >&2 + echo " $0 ${*}" >&2 + echo >&2 + echo " CMake caches it, so this is only needed once per build dir." >&2 + fi + exit 1 + fi fi echo "[home_install] building ..." cmake --build "$BUILD_DIR" @@ -63,6 +163,71 @@ fi echo "[home_install] installing to $FLM_PREFIX ..." cmake --install "$BUILD_DIR" --prefix "$FLM_PREFIX" +# ---- stage the rai (ryzenai-corelib) runtime ------------------------------- +# A corelib-backed build links libryzenai_corelib.so, which no install rule +# covers: CMake only globs lib/ for engine libraries, and corelib is +# not one of those. Copying it alone is still not enough -- it reaches +# libdyn_dispatch_core.so through an absolute RUNPATH into the out-of-tree +# DynamicDispatch prefix it was built against, so the install would break as +# soon as that directory moved. +# +# Copy corelib together with every non-system library it pulls in, leaving the +# prefix self-contained. The dynamic loader searches LD_LIBRARY_PATH (set by the +# env script below) before DT_RUNPATH, so these copies take precedence over the +# build-time paths baked into the binaries. +CACHE_FILE="$BUILD_DIR/CMakeCache.txt" +if [[ -f "$CACHE_FILE" ]] && grep -q '^FLM_ENABLE_RAI:BOOL=ON' "$CACHE_FILE"; then + # HRX puts engine libs in lib/flm; the portable and XRT layouts use lib/. + if grep -q '^FLM_USE_HRX:BOOL=ON' "$CACHE_FILE"; then + RAI_LIB_DEST="$FLM_PREFIX/lib/flm" + else + RAI_LIB_DEST="$FLM_PREFIX/lib" + fi + + CORELIB="$(sed -n 's/^RYZENAI_CORELIB_LIBRARY:FILEPATH=//p' "$CACHE_FILE")" + if [[ -z "$CORELIB" || ! -f "$CORELIB" ]]; then + echo "[home_install] ERROR: rai build, but RYZENAI_CORELIB_LIBRARY in" >&2 + echo " $CACHE_FILE does not point at a readable file." >&2 + exit 1 + fi + + echo "[home_install] rai build detected; staging corelib into $RAI_LIB_DEST" + mkdir -p "$RAI_LIB_DEST" + install -m 0755 "$CORELIB" "$RAI_LIB_DEST/" + # ldd resolves the whole transitive chain, so filtering out the system + # directories here leaves exactly the libraries that ship with the build. + DD_CORE_DIR="" + while read -r _soname _arrow dep _addr; do + case "$dep" in + /lib/*|/lib64/*|/usr/lib/*|/usr/lib64/*|"") continue ;; + esac + [[ -f "$dep" ]] || continue + if [[ "$(basename "$dep")" == libdyn_dispatch_core.so* ]]; then + DD_CORE_DIR="$(dirname "$dep")" + fi + echo "[home_install] + $(basename "$dep")" + install -m 0755 "$dep" "$RAI_LIB_DEST/" + done < <(ldd "$CORELIB" | grep '=>' || true) + + # libdyn_bins.so is invisible to the ldd walk above: nothing declares it as + # a DT_NEEDED. DynamicDispatch dlopen()s it at runtime, from the directory + # holding libdyn_dispatch_core.so (it dladdr()s itself to find that dir), + # and when the file is absent Transaction::load_large_txn_ops_dll() just + # returns -- no warning, no error. + # + # It only exists when DD was configured with DD_MDS_IN_BINS_DLL=ON, which + # moves every transaction/mds/*.elf -- i.e. the whole AIE4 kernel set -- out + # of the static transaction lib and into this library. Skipping it leaves + # the AIE4 shape table empty, and the first matmul dies with + # Target Shape (K: 3072, N: 3072, Gs: 64) ... not supported in this + # supported shape list. Shape list size: 0 + # which reads like an unsupported model rather than a missing file. + if [[ -n "$DD_CORE_DIR" && -f "$DD_CORE_DIR/libdyn_bins.so" ]]; then + echo "[home_install] + libdyn_bins.so (dlopen'd AIE4 kernel package)" + install -m 0755 "$DD_CORE_DIR/libdyn_bins.so" "$RAI_LIB_DEST/" + fi +fi + # ---- emit the environment script ------------------------------------------ ENV_SCRIPT="$FLM_PREFIX/flm_env.sh" echo "[home_install] writing env script: $ENV_SCRIPT" @@ -91,8 +256,13 @@ else echo "[flm_env] WARNING: \$XRT_DIR/setup.sh not found; falling back to LD_LIBRARY_PATH" >&2 export LD_LIBRARY_PATH="\$XRT_DIR/lib:\${LD_LIBRARY_PATH:-}" fi -# Belt-and-suspenders: also expose the bundled libs explicitly. -export LD_LIBRARY_PATH="\$FLM_PREFIX/lib/flm:\${LD_LIBRARY_PATH:-}" +# Belt-and-suspenders: also expose the bundled libs explicitly. Both engine +# directories are listed because the layout depends on the runtime backend +# (HRX uses lib/flm, XRT and portable builds use lib), and a rai build stages +# libryzenai_corelib.so plus its DynamicDispatch dependency into whichever one +# applies. This has to precede DT_RUNPATH, which still points at the machine +# the libraries were built on. +export LD_LIBRARY_PATH="\$FLM_PREFIX/lib:\$FLM_PREFIX/lib/flm:\${LD_LIBRARY_PATH:-}" # 3. Put flm on PATH. export PATH="\$FLM_PREFIX/bin:\$PATH" From 851dfe7fda93c319151cb4a2b93dc60828fc408b Mon Sep 17 00:00:00 2001 From: alfxu_amdeng Date: Wed, 23 Sep 2026 16:40:06 -0700 Subject: [PATCH 04/17] fix(rai): pin XILINX_XRT instead of letting XRT guess it XRT does not reach its own plugins through DT_NEEDED. It dlopens libxrt_core and the libxrt_driver_xdna NPU driver from a path it builds as $XILINX_XRT/lib/x86_64-linux-gnu/, and when XILINX_XRT is unset it guesses that root three directories above wherever libxrt_coreutil was loaded from. A second XRT anywhere on LD_LIBRARY_PATH aims the guess at a directory holding no plugins, and nothing says so: the NPU just comes up with no driver. corelib reports corelib: no AIE4 hw_context (unordered_map::at) and the direct open added in 8c38ad4f then fails too, with No such library '/home/alfxu/lib/x86_64-linux-gnu/libxrt_core.so.2' naming a directory that was never an XRT install. The rai build is the one that gets hit, because it bundles no XRT of its own. Resolve the root at install time instead, accepting a candidate only if the directory XRT will actually dlopen from exists, and export it from flm_env.sh so the guess never runs. An XRT that ships setup.sh still speaks for itself; the new path covers the distro layout, where the libraries sit under /usr and the old /opt/xilinx/xrt default resolved to an empty directory. Co-Authored-By: Claude Opus 5 --- src/home_install.sh | 57 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 54 insertions(+), 3 deletions(-) diff --git a/src/home_install.sh b/src/home_install.sh index b955094d8..2236657fd 100755 --- a/src/home_install.sh +++ b/src/home_install.sh @@ -45,6 +45,32 @@ PRESET="${PRESET_FROM_ENV:-linux-default}" # Where XRT (the AMD NPU runtime) is installed. Override if non-standard. XRT_DIR="${XRT_DIR:-/opt/xilinx/xrt}" +# XRT does not reach its own plugins through DT_NEEDED. It dlopens libxrt_core, +# the libxrt_driver_xdna NPU driver and the rest at run time from a path it +# builds as $XILINX_XRT/lib/x86_64-linux-gnu/, and when XILINX_XRT is unset +# it guesses that root three directories above wherever libxrt_coreutil happened +# to be loaded from. Any second copy of XRT on LD_LIBRARY_PATH sends the guess +# somewhere with no lib/x86_64-linux-gnu under it, and then the NPU comes up +# with no driver plugin: corelib reports "no AIE4 hw_context (unordered_map::at)" +# and even a bare xrt::device(0) fails with "No such library .../libxrt_core.so.2". +# That bites the rai build hardest, because it bundles no XRT of its own. +# +# So resolve the root here rather than leaving it to the guess, and accept a +# candidate only if the directory XRT will actually dlopen from exists -- that +# way a wrong answer surfaces at install time instead of at the first NPU call. +detect_xrt_root() { + local cand + for cand in "${XILINX_XRT:-}" "$XRT_DIR" /usr/local /usr; do + [[ -n "$cand" ]] || continue + if [[ -f "$cand/lib/x86_64-linux-gnu/libxrt_core.so.2" ]]; then + printf '%s\n' "$cand" + return 0 + fi + done + return 1 +} +XRT_ROOT="$(detect_xrt_root || true)" + DO_BUILD=1 WANT_RAI=0 for arg in "$@"; do @@ -229,6 +255,14 @@ if [[ -f "$CACHE_FILE" ]] && grep -q '^FLM_ENABLE_RAI:BOOL=ON' "$CACHE_FILE"; th fi # ---- emit the environment script ------------------------------------------ +if [[ -n "$XRT_ROOT" ]]; then + echo "[home_install] XRT runtime root: $XRT_ROOT" +else + echo "[home_install] WARNING: found no XRT install providing" >&2 + echo "[home_install] lib/x86_64-linux-gnu/libxrt_core.so.2; the NPU will be" >&2 + echo "[home_install] unavailable unless \$XRT_DIR/setup.sh supplies it." >&2 +fi + ENV_SCRIPT="$FLM_PREFIX/flm_env.sh" echo "[home_install] writing env script: $ENV_SCRIPT" cat > "$ENV_SCRIPT" < "$ENV_SCRIPT" <&2 - export LD_LIBRARY_PATH="\$XRT_DIR/lib:\${LD_LIBRARY_PATH:-}" + echo "[flm_env] WARNING: no XRT runtime found at install time and" >&2 + echo "[flm_env] \$XRT_DIR/setup.sh is missing; the NPU will be unavailable." >&2 fi # Belt-and-suspenders: also expose the bundled libs explicitly. Both engine # directories are listed because the layout depends on the runtime backend From 1efbdd26fdc8e76fc3cdfbd1b9548eec7ae283fd Mon Sep 17 00:00:00 2001 From: alfxu_amdeng Date: Wed, 23 Sep 2026 17:12:10 -0700 Subject: [PATCH 05/17] fix(rai): give the engines a real device, and say why when there is none Three faults on the way from "the NPU did not come up" to something a user can act on. The engines were handed nullptr. corelib holds the device, so a rai build has to pass corelib's own xrt::device down rather than the null one the frontend starts with. When the runtime failed to come up at all, the reason went to DO_VERBOSE. VERBOSE is a compile-time macro, so on a release build those lines are compiled out and no flag brings them back; what the user gets instead is [ERROR] NPU device instance is nullptr raised much later, from whichever model first reaches for the device -- a symptom several steps from the cause. Keep the reason in a string and report it once, up front, for the commands that cannot run without a device. And corelib having no device took down far more than rai. It gets its device from a bare AIE4 hw_context; when that cannot be created, DD swallows the XRT error in a catch(...) and then throws unordered_map::at on the key it never inserted, corelib reports "no AIE4 hw_context", and get_device() goes null for the whole process. But a build with FLM_ENABLE_RAI still serves every other family through the flm backend, which opens its own device and never touches corelib. Open a device directly when corelib has none, and say so: [FLM] rai backend unavailable (corelib unavailable: corelib has no device context); opened the NPU directly This does not bring back the two-device bug, where a buffer created against a second xrt::device for the same NPU bound without error and then never completed: that needed corelib live and holding a device of its own, and the fallback only runs when it has none. Co-Authored-By: Claude Opus 5 --- src/include/rai/corelib_device.hpp | 42 +++++++++----- src/src/main.cpp | 91 +++++++++++++++++++++++++----- 2 files changed, 107 insertions(+), 26 deletions(-) diff --git a/src/include/rai/corelib_device.hpp b/src/include/rai/corelib_device.hpp index 557f52298..667477d09 100644 --- a/src/include/rai/corelib_device.hpp +++ b/src/include/rai/corelib_device.hpp @@ -1,13 +1,12 @@ /// \file corelib_device.hpp -/// \brief Access to the xrt::device that ryzenai-corelib owns. +/// \brief Access to the NPU device that ryzenai-corelib owns. /// \note FastFlowLM consumes corelib through the C ABI in -/// (see corelib_api.hpp), which has no device accessor: the device lives -/// behind corelib's C++ entry point ryzenai::corelib::GetDevice(). The -/// declaration is reproduced here rather than pulled from a corelib C++ -/// header so that this tree keeps depending on exactly one corelib header; -/// it resolves at link time against the statically linked corelib -/// (RYZENAI_CORELIB_STATIC), so a signature drift is a link error, not a -/// silent mismatch. +/// (see corelib_api.hpp), and ryzenai_corelib_get_device() is the device +/// accessor in it. corelib's own ryzenai::corelib::GetDevice() is an +/// inline wrapper around that same call rather than an exported entry +/// point, so there is nothing to link against and no reason to reach for +/// the C++ layer: going through CorelibApi keeps every corelib call in +/// this tree on one resolved function table. #pragma once #if defined(FLM_USE_HRX) @@ -15,11 +14,28 @@ #endif #include "device_runtime.hpp" +#include "rai/corelib_runtime.hpp" -namespace ryzenai::corelib { +namespace flm::corelib { -/// \brief the device corelib initialized; valid until ryzenai_corelib_cleanup() -/// \return corelib's device, shared with every FLM engine -const xrt::device& GetDevice(); +/// \brief the device corelib dispatches on, shared with every engine here +/// \param runtime an initialized corelib runtime +/// \return corelib's device, or nullptr when this machine has no NPU +/// \note Valid until ryzenai_corelib_cleanup(); the process runtime outlives +/// every use of the returned pointer. +/// \note flm_rt is an alias for xrt on this path, so corelib's device already +/// has the type the engines take. corelib hands it out const and the +/// engines want it mutable; the constness is cast away rather than a +/// second device opened, because a buffer object created against a +/// different xrt::device for the same NPU binds without error and then +/// never completes. +inline flm_rt::device* SharedDevice(const CorelibRuntime& runtime) noexcept { + const void* device = runtime.api()->functions().get_device(); + if (device == nullptr) { + return nullptr; + } + return const_cast( + reinterpret_cast(device)); +} -} // namespace ryzenai::corelib +} // namespace flm::corelib diff --git a/src/src/main.cpp b/src/src/main.cpp index b66d2a96a..061e5a05c 100644 --- a/src/src/main.cpp +++ b/src/src/main.cpp @@ -475,27 +475,30 @@ struct RaiProcessGuard { } } }; -#else +#endif + ///@brief open the NPU device that every engine in this process shares +///@param why if non-null, receives why the device could not be opened ///@return the shared device, or nullptr when no NPU could be opened ///@note Function-local static, so the lifetime is tied to the process exactly as /// the per-Runner devices used to be. A machine with no NPU must still be /// able to run `flm list`/`pull`/`version`, so failure is a null pointer, /// not an error. -static flm_rt::device* open_npu_device() { +static flm_rt::device* open_npu_device(std::string* why) { try { static flm_rt::device npu_device = flm_rt::device(0); return &npu_device; } catch (const std::exception& e) { + if (why) *why = e.what(); DO_VERBOSE(1, { header_print("FLM", "No NPU device available: " << e.what()); }); return nullptr; } catch (...) { + if (why) *why = "unknown error"; return nullptr; } } -#endif ///@brief main function ///@param argc the number of arguments @@ -561,21 +564,67 @@ int main(int argc, char* argv[]) { // Declared before anything that can return so the destructor covers the // early exits and the catch-all below. RaiProcessGuard rai_guard; + // Declared out here so the early exits below still see a device, and so a + // corelib that fails to come up leaves it null rather than undefined. + flm_rt::device* npu_device = nullptr; + // Kept so the reason survives into a release build. VERBOSE is a + // compile-time macro, so every DO_VERBOSE below is compiled out and a user + // is left with "NPU device instance is nullptr" -- raised much later, by + // whichever model reaches for the device first -- and nothing to act on. + std::string npu_open_error; + // Set when corelib failed but a direct device was opened anyway: the rai + // backend is gone, the rest of the build is not. + std::string corelib_note; try { - flm::corelib::CorelibRuntime::GetOrCreate(std::filesystem::path(exe_dir)); + const auto runtime = + flm::corelib::CorelibRuntime::GetOrCreate(std::filesystem::path(exe_dir)); + // corelib opens the NPU for this process, so take its device rather + // than opening a second one: a buffer object created against a + // different xrt::device for the same NPU binds without error and then + // never completes. GetOrCreate holds the runtime process-wide, so the + // device stays valid until RaiProcessGuard tears it down at exit. + npu_device = flm::corelib::SharedDevice(*runtime); + if (npu_device == nullptr) { + npu_open_error = + "corelib started but reports no NPU device on this machine"; + DO_VERBOSE(1, { + header_print("FLM", "corelib reports no NPU device on this machine"); + }); + } } catch (const std::exception& e) { + // A box with no NPU must still run `flm list`/`pull`/`version`, so this + // stays a null device rather than an error, as in the non-rai path. + npu_open_error = std::string("corelib unavailable: ") + e.what(); DO_VERBOSE(1, { header_print("FLM", "corelib unavailable: " << e.what()); }); } - // TODO: FIXME - corelib's device is not a drop-in for the flm device the - // AutoModel engines expect; the two ownership models conflict, so there is - // no supported way to hand corelib's device out here yet. Until that is - // resolved a rai build reports no flm device, which is harmless because - // the rai backend does not drive the NPU through one. - // flm_rt::device* npu_device = - // const_cast(&ryzenai::corelib::GetDevice()); - flm_rt::device* npu_device = nullptr; + if (npu_device == nullptr) { + // corelib having no device is a reason for the rai backend to be + // unavailable, not for the process to have no NPU at all. Every other + // family is served by the flm backend, which opens its own device and + // never touches corelib -- so opening one directly keeps those models + // working instead of failing the whole process over a backend they do + // not use. On a box whose NPU has no creatable AIE4 hw_context this is + // the difference between "rai is unavailable" and "flm has no NPU". + // + // This does not bring back the two-device bug, where a buffer created + // against a second xrt::device for the same NPU bound without error and + // then never completed. That needed corelib to be live and holding a + // device of its own; here it has none, so the process still ends up + // with exactly one. + corelib_note = npu_open_error; + std::string raw_error; + npu_device = open_npu_device(&raw_error); + if (npu_device != nullptr) { + npu_open_error.clear(); + } else { + npu_open_error += "; opening the NPU directly also failed: " + raw_error; + corelib_note.clear(); + } + } #else - flm_rt::device* npu_device = open_npu_device(); + std::string npu_open_error; + std::string corelib_note; + flm_rt::device* npu_device = open_npu_device(&npu_open_error); #endif // Which generation this binary is for is decided by FLM_ENABLE_RAI at @@ -594,6 +643,22 @@ int main(int argc, char* argv[]) { header_print("FLM", "NPU platform: " << utils::platform_id(platform)); } + // Say once, here, why there is no device. The commands below cannot run + // without one, and the message they eventually produce names the symptom + // rather than the cause: it is raised from whichever model first reaches + // for the device, long after the runtime that failed to come up. + if (print_status && needs_npu && !corelib_note.empty()) { + header_print("FLM", "rai backend unavailable (" + << corelib_note + << "); opened the NPU directly"); + } + if (needs_npu && npu_device == nullptr) { + header_print("ERROR", "No NPU device available" + << (npu_open_error.empty() + ? std::string() + : ": " + npu_open_error)); + } + // The rai build of phi4-mini-it installs under its own directory name, so // on any other platform that directory is no longer reachable by a tag and // `flm remove` cannot clean it up. Point it out; never delete it. From 40aa9fe47c1e8b94543081b2ae1b2adbdc9affa6 Mon Sep 17 00:00:00 2001 From: alfxu_amdeng Date: Thu, 24 Sep 2026 10:06:51 -0700 Subject: [PATCH 06/17] feat(rai): decide what an install can run from two independent keys Adding the corelib backend took the FastFlowLM models away: the catalog had one notion of "supported", the build had another, and whichever was asked last won. The fix took several passes -- a per-flavour catalog, a backend key on every entry -- and this is where they landed, as one change, because the intermediate shapes were never true of the tree for long enough to be worth bisecting through. Two questions are asked of every catalog entry, and they are independent. Which silicon the entry's artifacts were built for is "supported_platforms", aie2p or aie_next, checked against utils::get_device(). Every one of the 42 FastFlowLM entries names aie2p; the corelib one names aie_next. A missing key still means every generation, so a catalog written before the key keeps working. Which kernel flow they need is the family name: a family ending in -rai is ryzenai-corelib's, everything else is FastFlowLM's, checked against the flows this build actually linked. The flow is derived, never authored -- model_list stamps "backend" onto the entry from the tag it was found under, which is where AutoModel already looks, so a hand-written one would only be a second place to be wrong. --backend and FLM_BACKEND still win over it. An entry failing either is dropped before anything downstream sees it, because a model that is listed but cannot run is worse than one that is not listed. The generation is asked first and of everyone: linking corelib says what kernels the binary has and nothing about the silicon it is on, so a corelib build does not get to see a corelib model merely by existing. That makes the corelib phi4 its own top-level family, phi4-mini-it-rai, beside phi4-mini-it rather than patching it through platform_overrides. Two packages that share nothing but a lineage -- a Q8_0 GGUF against an NPU2/Q4NX build -- now read as two entries in one model_list.json, the merge-patch machinery is gone, and the separate tag means model_info.json needs no model_info_key redirect to reach the corelib records. get_device() is a stand-in that consults nothing at all: not the environment, not FLM_ENABLE_RAI, just default_npu_platform(). One constant is the whole answer, so there is exactly one thing for the real probe to replace. It is aie_next for now, while corelib is what is being brought up -- which means a build without corelib lists no models, since every FastFlowLM entry names aie2p. That state is reached on purpose and is not fatal: model_list names the generation and the linked kernels in one line, then each command fails on its own terms, and `flm --help` still works. Setting the constant back to aie2p restores the 42. Co-Authored-By: Claude Opus 5 --- docs/docs/instructions/cli.md | 2 +- docs/docs/models/phi.md | 6 +- src/common/AutoModel/automodel.cpp | 11 +- src/common/AutoModel/model_backend.cpp | 24 +- src/common/models/README.md | 106 +++-- src/common/npu_platform.cpp | 14 +- src/home_install.sh | 144 +++++-- src/include/AutoModel/model_backend.hpp | 67 +-- src/include/model_list.hpp | 134 ++++-- src/include/utils/npu_platform.hpp | 61 +-- src/include/utils/utils.hpp | 4 - src/model_list.json | 164 +++++--- src/src/main.cpp | 35 +- src/test/model_backend/test_model_backend.cpp | 57 ++- .../test_model_list_platform.cpp | 386 ++++++++++++------ src/test/phi4_rai/test_model_downloader.cpp | 54 ++- 16 files changed, 837 insertions(+), 432 deletions(-) diff --git a/docs/docs/instructions/cli.md b/docs/docs/instructions/cli.md index e8cc76a76..32512b882 100644 --- a/docs/docs/instructions/cli.md +++ b/docs/docs/instructions/cli.md @@ -241,7 +241,7 @@ flm serve llama3.2:1b --ctx-len 8192 | `flm` | FastFlowLM's own NPU kernels | Strix / Krackan Point | | `rai` | AMD's ryzenai-corelib | the next NPU generation | -You normally never set this. A build links one kernel flow — `FLM_ENABLE_RAI` selects `rai`, otherwise `flm` — and that is the default. `flm run` separately prints the silicon it was built for as `NPU platform: stx`. A model family has at most one engine per backend, so there is nothing to choose between. +You normally never set this. The model tag already names the flow — a family ending in `-rai` wants corelib's kernels, everything else FastFlowLM's — and `flm list` only ever offers you tags this build and this machine can actually run. `flm run` separately prints the silicon as `NPU platform: aie_next`. A model family has at most one engine per backend, so there is nothing to choose between. The flag exists for overriding the detection, and for the targets that will join this list later: diff --git a/docs/docs/models/phi.md b/docs/docs/models/phi.md index c6dee6ac6..3f366dd56 100644 --- a/docs/docs/models/phi.md +++ b/docs/docs/models/phi.md @@ -26,7 +26,7 @@ flm run phi4-mini-it:4b ## 🧪 Model Card: Phi-4-mini-instruct on the rai backend (developer preview) -- **Tag:** `phi4-mini-it:4b` — the same tag as the NPU2 build. A build targets one NPU generation (`FLM_ENABLE_RAI` selects `aie_next`, otherwise `stx` / Strix / Krackan Point), and the tag resolves to the artifacts that generation can run. There is no separate tag for it; `flm list` on a rai build shows only the models it can run. +- **Tag:** `phi4-mini-it-rai:4b` — its own tag, beside `phi4-mini-it:4b`, which stays the NPU2/Q4NX build. They are two entries in the one catalog, `model_list.json`, describing two different packages, and nothing about one is a patch on the other. Two things decide whether an install offers this one, and **the first is the machine**: `supported_platforms` says `aie_next` here and `aie2p` there, checked against the generation FastFlowLM reads from the device, so on shipping silicon this tag is not listed even by a build that has corelib compiled in. Only then does the second apply — whether corelib's kernels were linked at all, which the `-rai` suffix on the family name is what asks for. `utils::get_device()` is still a stand-in and currently answers `aie_next` for every build, so a corelib developer build offers this tag and not the NPU2 one; a build without corelib on the same setting offers no models at all, and says so. - **Backend:** `rai` — the backend names the kernel provider; FastFlowLM reaches these kernels through AMD's `ryzenai_corelib` - **Source format:** GGUF, read directly. No ONNX model, no tensor manifest, and no converted or packed weight file is produced or shipped. - **Quantization:** GGML `Q8_0` in the file, requantized to **group 64** while the weights are packed for the device, through corelib's explicit `*_create_gguf_requantized` entry points. This is a **lossy** second quantization step and it is not reversible; output will differ from the Q8_0 source. @@ -91,12 +91,12 @@ Phi-4-mini itself supports 128k, and the existing `phi4-mini-it:4b` tag defaults ### No fallback -Backend selection follows the build, never a filename or a quantization level. What this binary was built for decides two things: *which catalog entry* the tag resolves to — the NPU2/Q4NX entry on `stx`, this one on `aie_next` — and which backend runs it, since a build links exactly one kernel flow. Once this entry is selected, there is no fallback: if corelib is missing, unloadable, or the wrong version, the tag **fails to load with a diagnostic** rather than quietly running on CPU or on the NPU2/Q4NX backend. +Backend selection follows the tag, never a filename or a quantization level. The `-rai` suffix *is* the request for corelib's kernels, and the entry's `supported_platforms` decides whether this machine is offered it at all. There is nothing to fall back to and nothing to resolve between: `phi4-mini-it:4b` and `phi4-mini-it-rai:4b` are separate tags, so asking for one never gets you the other. Once this tag is selected, there is no fallback: if corelib is missing, unloadable, or the wrong version, the tag **fails to load with a diagnostic** rather than quietly running on CPU or on the NPU2/Q4NX backend. ### Naming the backend yourself The two engines are registered under the kernel provider they use, `flm` and `rai`, and you can name one with `--backend`, with `FLM_BACKEND`, or with a `"backend"` field on an `/api/chat` or `/api/generate` request. The [CLI reference](../instructions/cli.md) has the full precedence table. -This does not widen what the hardware can run. A given release is built for one NPU generation, so only that generation's backend is compiled in; asking for the other one fails immediately, naming what this build actually has, instead of failing deep inside an engine that was never going to work. If what you meant was the other catalog entry, that is a different build of FLM, not a different flag. +This does not widen what the hardware can run. A given release is built for one NPU generation, so only that generation's backend is compiled in; asking for the other one fails immediately, naming what this build actually has, instead of failing deep inside an engine that was never going to work. If what you meant was the other package, ask for its tag — and if the tag is not listed, that is a different build of FLM, not a different flag. --- \ No newline at end of file diff --git a/src/common/AutoModel/automodel.cpp b/src/common/AutoModel/automodel.cpp index 0c821872c..81559a9e1 100644 --- a/src/common/AutoModel/automodel.cpp +++ b/src/common/AutoModel/automodel.cpp @@ -205,11 +205,14 @@ void AutoModel::_shared_load_backend(std::string model_path, json model_info, auto& registry = flm::backend::BackendRegistry::instance(); std::string source; - // A build links one kernel flow, so that is the default when neither - // --backend nor FLM_BACKEND says otherwise. + // The entry carries the kernel flow its artifacts were built for -- the + // weights differ between flows, so this belongs to the model rather than + // to the build. model_list wrote it there from the tag the entry was + // found under; --backend and FLM_BACKEND still win over it. + const std::string catalog_backend = model_info.value( + flm::backend::kBackendKey, std::string(flm::backend::kFlmBackendId)); const std::string id = flm::backend::resolve_backend_id( - family, flm::backend::build_default_backend_id(), requested_backend, - &source); + family, catalog_backend, requested_backend, &source); // Same model on the same backend is a no-op; a different backend is a real // reload even when the path has not changed. diff --git a/src/common/AutoModel/model_backend.cpp b/src/common/AutoModel/model_backend.cpp index 62ea33a90..b5c3ea30d 100644 --- a/src/common/AutoModel/model_backend.cpp +++ b/src/common/AutoModel/model_backend.cpp @@ -5,6 +5,7 @@ #include "utils/utils.hpp" #include +#include #include #include #include @@ -109,6 +110,19 @@ bool BackendRegistry::has(const std::string& family, return per_family != factories_.end() && per_family->second.count(id) != 0; } +std::vector BackendRegistry::backend_ids() const { + std::lock_guard lock(mutex_); + std::set ids; + for (const auto& [family, per_family] : factories_) { + (void)family; + for (const auto& [id, entry] : per_family) { + (void)entry; + ids.insert(id); + } + } + return {ids.begin(), ids.end()}; +} + std::unique_ptr BackendRegistry::create( const std::string& family, const std::string& id, const BackendContext& context) const { @@ -129,15 +143,15 @@ std::string resolve_backend_id(const std::string& family, chosen_source = "FLM_BACKEND"; } else { chosen = fallback; - chosen_source = "build default"; + chosen_source = "model catalog"; } - if (!BackendRegistry::instance().has(family, chosen)) { + auto& registry = BackendRegistry::instance(); + if (!registry.has(family, chosen)) { throw std::runtime_error( "Backend '" + chosen + "' (from " + chosen_source + - ") is not compiled into this build of flm. Model family '" + - family + "' provides: " + - Join(BackendRegistry::instance().available(family))); + ") is not available for model family '" + family + + "'. It provides: " + Join(registry.available(family))); } if (source) *source = chosen_source; diff --git a/src/common/models/README.md b/src/common/models/README.md index e24929bfa..16a47b971 100644 --- a/src/common/models/README.md +++ b/src/common/models/README.md @@ -6,11 +6,23 @@ only model on this path today, so every file named below has a `phi4` counterpar you can read straight through. A backend names *where the kernels come from*: `flm` is FastFlowLM's own kernel -flow, `rai` is corelib. That is a separate axis from which silicon a build -targets, which is `utils::npu_platform` (`stx`, `aie_next`). The two line up -one-to-one today — a rai build is an aie_next build — but they are different -questions and they get different names, so that the day they stop lining up is -not the day every string in the tree starts lying. +flow, `rai` is corelib. That is a separate axis from which silicon the host has, +which is `utils::npu_platform` (`aie2p`, `aie_next`) and is answered at run time +by `utils::get_device()`. They are genuinely independent: a build links every +flow it was configured with, and several flows can serve one generation. Keying +either axis off the other is how adding corelib once took the FastFlowLM models +away. + +`get_device()` is a **stand-in** until the real probe lands, and it consults +nothing: it returns `default_npu_platform()`. Not the environment, and *not* +whether corelib was linked — linking corelib says what kernels this binary has, +never what silicon it is running on, and an install that lists a model it +cannot run is worse than one that lists nothing. So the constant *is* the +answer, for every build: change it and rebuild to move a whole install to the +other generation, and the probe replaces the one function. + +It is **`aie_next`** today, while the corelib path is being brought up. Read +the next paragraph before being surprised by what `flm list` shows. This is a contributor document. For *using* a backend once it exists — `--backend`, `FLM_BACKEND`, precedence — see [`docs/docs/instructions/cli.md`](../../../docs/docs/instructions/cli.md). @@ -241,48 +253,69 @@ Also register the FLM-side engine if the family has one: ## 6. The catalog -Two files, and both must agree. +Two files, and both of them must agree. -**[`model_list.json`](../../model_list.json)** — the entry. Phi-4 shares one tag -across both NPU generations, so the aie_next artifacts arrive as a -`platform_overrides.aie_next` patch: +**[`model_list.json`](../../model_list.json)** is the one catalog. Every install +ships it and every build reads it; what differs between installs is how much of +it survives [`model_list::apply_support_filter`](../../include/model_list.hpp), +which drops every entry this machine or this build cannot run. A model that is +offered nowhere is the same to a user as a model that does not exist, so both +axes prune: + +| the entry is dropped when | said by | +|---|---| +| the host is not silicon the entry names | `"supported_platforms"` on the entry | +| the build did not link the kernel flow the entry needs | the **family name**: a family ending in `-rai` is corelib's, everything else is FastFlowLM's | + +So the corelib flavour of a model is **its own top-level family**, named +`-rai`, sitting beside the stock one rather than patching it: ```jsonc -"": { +"-rai": { "": { - "supported_platforms": ["stx", "aie_next"], - "platform_overrides": { - "aie_next": { - "name": "", - "url": "...", "file_url": "...", "size": 4100140571, - "default_context_length": 4096, - "files": [".gguf", "tokenizer.json", "tokenizer_config.json", "config.json"], - "file_sources": { "tokenizer.json": { "url": "...", "revision": "..." } }, - "model_info_key": "-rai:", - "ms_url": null - } - } + "supported_platforms": ["aie_next"], + "name": "", + "url": "...", "file_url": "...", "size": 4100140571, + "default_context_length": 4096, + "files": [".gguf", "tokenizer.json", "tokenizer_config.json", "config.json"], + "file_sources": { "tokenizer.json": { "url": "...", "revision": "..." } } } } ``` Points that are easy to get wrong: -- The patch is a **JSON merge-patch**: arrays replace wholesale, and `null` - *deletes* a key — that is what `"ms_url": null` is doing. -- `supported_platforms` is pruned at load, and **only aie_next support needs a - tag**: an entry that omits the key is stx-only, which is the overwhelming - majority. Do not write `["stx"]`; it restates the default. -- **The entry names no backend.** There is no `supported_backends` and no - `details.execution_backend` — both are retired. By the time an entry reaches - the loader it has already been filtered to the platform this build targets, - and the build links exactly one kernel flow, so there is nothing left for the - entry to decide. `--backend` and `FLM_BACKEND` override that default, and are - checked against the registry, not against the entry. +- **The family name is the mechanism.** Nothing in the file says `rai`; + `model_list` reads the `-rai` suffix off the tag and stamps the answer onto + the entry as `"backend"`, which is what + [`AutoModel`](../AutoModel/automodel.cpp) later reads. Do not write `backend` + by hand — it is derived, and a hand-written one is overwritten. + `--backend` and `FLM_BACKEND` still override it, and are checked against the + registry. The plural `supported_backends` and `details.execution_backend` are + retired, as are `supported_backend` and `platform_overrides`. +- **`supported_platforms` is required on every shipped entry**, and + `src/test/model_list_platform` fails if one is missing or disagrees with its + family name. Omitting it is legal — it means *every* generation, which is what + a catalog written before the key meant — but a shipped entry should say what + it was built for. All 42 FastFlowLM entries are `["aie2p"]`; the corelib one + is `["aie_next"]`. +- Separate families mean **separate tags**, so no `model_info_key` redirect is + needed: `model_info.json` keys the corelib records under `-rai:` + directly, which is also the tag a user types. - `file_sources` pins a per-file origin + revision when the weights and the tokenizer come from different repos (very common with GGUF mirrors). -- `model_info_key` redirects the downloader to a differently-named record set, - which is needed exactly because the tag is shared across platforms. +- The two keys are checked independently, and **the platform is checked first + and for everyone**. On today's `aie_next` default that means a corelib build + offers `phi4-mini-it-rai` and nothing else, and a build *without* corelib + offers **nothing at all** — the 42 FastFlowLM entries are pruned by the + generation, and the corelib entry by the kernels it would need. That is the + cost of bringing up the next generation before its probe exists, and it is + reversed by setting `default_npu_platform()` back to `aie2p`, which gives + both builds the same 42 tags and neither the corelib one. +- An empty catalog is therefore an ordinary state, not a crash: `model_list` + prints one line naming the generation and the linked kernels, `flm list` says + it found nothing, `flm run` says the tag is not found, and `flm --help` still + works. A build that aborted here could not even tell you why. **[`model_info.json`](../../model_info.json)** — one record per file, with `size` and `sha256`. The downloader refuses anything it cannot match @@ -349,5 +382,6 @@ Check, in order: - [ ] `BackendTraits` is `inline` in the header - [ ] registered in `builtin_backends.cpp` under `#if defined(FLM_ENABLE_RAI)` - [ ] no `#if FLM_ENABLE_RAI` anywhere in the frontend -- [ ] `model_list.json` + `model_info.json` agree, including `model_info_key` +- [ ] `model_list.json` and `model_info.json` agree, the corelib family is + named `-rai` and its `supported_platforms` says `aie_next` - [ ] frozen headers untouched diff --git a/src/common/npu_platform.cpp b/src/common/npu_platform.cpp index 8082c09bb..90bb86cf6 100644 --- a/src/common/npu_platform.cpp +++ b/src/common/npu_platform.cpp @@ -1,11 +1,21 @@ /// \file npu_platform.cpp -/// \brief the NPU generation this build targets (stx vs aie_next) +/// \brief which NPU generation this host has (aie2p vs aie_next) #include "utils/npu_platform.hpp" namespace utils { +npu_platform get_device() { + // No device read here: the probe that tells the generations apart is not in + // this tree, and nothing else in the build is allowed to stand in for it. + // Linking corelib in particular is not evidence -- which kernels were + // compiled is a fact about the build, the generation is a fact about the + // machine, and letting the first answer the second is how the two axes get + // quietly welded back together. + return default_npu_platform(); +} + std::optional parse_platform(std::string_view text) { - if (text == platform_id(npu_platform::stx)) return npu_platform::stx; + if (text == platform_id(npu_platform::aie2p)) return npu_platform::aie2p; if (text == platform_id(npu_platform::aie_next)) return npu_platform::aie_next; return std::nullopt; } diff --git a/src/home_install.sh b/src/home_install.sh index 2236657fd..daeacf64f 100755 --- a/src/home_install.sh +++ b/src/home_install.sh @@ -49,15 +49,17 @@ XRT_DIR="${XRT_DIR:-/opt/xilinx/xrt}" # the libxrt_driver_xdna NPU driver and the rest at run time from a path it # builds as $XILINX_XRT/lib/x86_64-linux-gnu/, and when XILINX_XRT is unset # it guesses that root three directories above wherever libxrt_coreutil happened -# to be loaded from. Any second copy of XRT on LD_LIBRARY_PATH sends the guess -# somewhere with no lib/x86_64-linux-gnu under it, and then the NPU comes up -# with no driver plugin: corelib reports "no AIE4 hw_context (unordered_map::at)" -# and even a bare xrt::device(0) fails with "No such library .../libxrt_core.so.2". -# That bites the rai build hardest, because it bundles no XRT of its own. +# to be loaded from. A rai install bundles no XRT for that guess to land on, so +# it names the root instead, and accepts a candidate only if the directory XRT +# will actually dlopen from exists -- that way a wrong answer surfaces at +# install time instead of at the first NPU call. # -# So resolve the root here rather than leaving it to the guess, and accept a -# candidate only if the directory XRT will actually dlopen from exists -- that -# way a wrong answer surfaces at install time instead of at the first NPU call. +# A stock install is left to the guess, exactly as it always was: its prefix +# carries its own XRT in the layout XRT expects, and the guess lands on it. +# Naming a root there, or putting a second XRT ahead of it on LD_LIBRARY_PATH, +# is what made a stock prefix report "No such library +# /home/$USER/lib/x86_64-linux-gnu/libxrt_core.so.2" and then "No such device +# with index '0'" on a machine whose NPU was fine. detect_xrt_root() { local cand for cand in "${XILINX_XRT:-}" "$XRT_DIR" /usr/local /usr; do @@ -69,7 +71,6 @@ detect_xrt_root() { done return 1 } -XRT_ROOT="$(detect_xrt_root || true)" DO_BUILD=1 WANT_RAI=0 @@ -88,6 +89,11 @@ if [[ "$WANT_RAI" -eq 1 && -z "$PRESET_FROM_ENV" ]]; then PRESET="linux-rai-on" fi +# Empty unless this is a rai install; see detect_xrt_root above for why only +# that half wants an answer. +XRT_ROOT="" +[[ "$WANT_RAI" -eq 1 ]] && XRT_ROOT="$(detect_xrt_root || true)" + # A rai install and a stock one differ in the binary and in the libraries staged # beside it, but occupy identical paths, so sharing a prefix means whichever ran # last silently replaces the other. Giving rai its own default keeps both usable @@ -189,6 +195,11 @@ fi echo "[home_install] installing to $FLM_PREFIX ..." cmake --install "$BUILD_DIR" --prefix "$FLM_PREFIX" +# An earlier split shipped a second catalog, model_list_rai.json, beside this +# one; a prefix that still holds it would answer "what can this install run" +# twice, with the stale copy winning nothing but confusion. +rm -f "$FLM_PREFIX/share/flm/model_list_rai.json" + # ---- stage the rai (ryzenai-corelib) runtime ------------------------------- # A corelib-backed build links libryzenai_corelib.so, which no install rule # covers: CMake only globs lib/ for engine libraries, and corelib is @@ -201,8 +212,22 @@ cmake --install "$BUILD_DIR" --prefix "$FLM_PREFIX" # prefix self-contained. The dynamic loader searches LD_LIBRARY_PATH (set by the # env script below) before DT_RUNPATH, so these copies take precedence over the # build-time paths baked into the binaries. +# +# Keyed on --rai rather than on the cache: a stock install has to be the stock +# install main ships, and reading the flag out of whichever build tree happened +# to be lying around made a plain ./home_install.sh drop libryzenai_corelib.so +# and a 400 MB libdyn_dispatch_core.so into ~/flm_exe, where the bundled XRT +# then had company it could not cope with. A tree that disagrees with the mode +# gets a warning instead. CACHE_FILE="$BUILD_DIR/CMakeCache.txt" -if [[ -f "$CACHE_FILE" ]] && grep -q '^FLM_ENABLE_RAI:BOOL=ON' "$CACHE_FILE"; then +if [[ "$WANT_RAI" -eq 0 ]]; then + if [[ -f "$CACHE_FILE" ]] && grep -q '^FLM_ENABLE_RAI:BOOL=ON' "$CACHE_FILE"; then + echo "[home_install] WARNING: $BUILD_DIR was configured with FLM_ENABLE_RAI=ON," >&2 + echo "[home_install] but this is a stock install: no corelib runtime is" >&2 + echo "[home_install] staged and the env script is the stock one. Re-run" >&2 + echo "[home_install] with --rai for a corelib install." >&2 + fi +elif [[ -f "$CACHE_FILE" ]] && grep -q '^FLM_ENABLE_RAI:BOOL=ON' "$CACHE_FILE"; then # HRX puts engine libs in lib/flm; the portable and XRT layouts use lib/. if grep -q '^FLM_USE_HRX:BOOL=ON' "$CACHE_FILE"; then RAI_LIB_DEST="$FLM_PREFIX/lib/flm" @@ -252,35 +277,36 @@ if [[ -f "$CACHE_FILE" ]] && grep -q '^FLM_ENABLE_RAI:BOOL=ON' "$CACHE_FILE"; th echo "[home_install] + libdyn_bins.so (dlopen'd AIE4 kernel package)" install -m 0755 "$DD_CORE_DIR/libdyn_bins.so" "$RAI_LIB_DEST/" fi +else + echo "[home_install] ERROR: --rai, but $BUILD_DIR was not configured with" >&2 + echo " FLM_ENABLE_RAI=ON, so there is no corelib runtime to" >&2 + echo " stage. Drop --no-build, or point BUILD_DIR at a rai tree." >&2 + exit 1 fi # ---- emit the environment script ------------------------------------------ -if [[ -n "$XRT_ROOT" ]]; then - echo "[home_install] XRT runtime root: $XRT_ROOT" -else - echo "[home_install] WARNING: found no XRT install providing" >&2 - echo "[home_install] lib/x86_64-linux-gnu/libxrt_core.so.2; the NPU will be" >&2 - echo "[home_install] unavailable unless \$XRT_DIR/setup.sh supplies it." >&2 +if [[ "$WANT_RAI" -eq 1 ]]; then + if [[ -n "$XRT_ROOT" ]]; then + echo "[home_install] XRT runtime root: $XRT_ROOT" + else + echo "[home_install] WARNING: found no XRT install providing" >&2 + echo "[home_install] lib/x86_64-linux-gnu/libxrt_core.so.2; the NPU will be" >&2 + echo "[home_install] unavailable unless \$XRT_DIR/setup.sh supplies it." >&2 + fi fi -ENV_SCRIPT="$FLM_PREFIX/flm_env.sh" -echo "[home_install] writing env script: $ENV_SCRIPT" -cat > "$ENV_SCRIPT" <&2 + export LD_LIBRARY_PATH="$XRT_DIR/lib:${LD_LIBRARY_PATH:-}" +fi +# Belt-and-suspenders: also expose the bundled libs explicitly. +export LD_LIBRARY_PATH="$FLM_PREFIX/lib/flm:${LD_LIBRARY_PATH:-}" +STOCK_ENV +)" +fi + +ENV_SCRIPT="$FLM_PREFIX/flm_env.sh" +echo "[home_install] writing env script: $ENV_SCRIPT" +cat > "$ENV_SCRIPT" < are prebuilt against causal_lm.hpp, so that header is /// a frozen ABI: adding or reordering a virtual there would silently shift @@ -34,29 +37,30 @@ class npu_xclbin_manager; namespace flm::backend { /// \brief FastFlowLM's own NPU kernel flow -/// \note Every model family registers this one; it is what a build without -/// FLM_ENABLE_RAI runs, on whatever generation it was built for. +/// \note Every model family registers this one, in every build, which is what +/// makes it the backend a catalog entry gets when it names none. inline constexpr const char* kFlmBackendId = "flm"; /// \brief kernels reached through ryzenai-corelib /// \note Only compiled in when FLM_ENABLE_RAI is on, and only for families that -/// have a corelib engine. +/// have a corelib engine. It is added to a build, never swapped in: a rai +/// build still runs every flm model with the flm kernels. inline constexpr const char* kRaiBackendId = "rai"; -/// \brief the backend a build falls back to when nothing else picks one -/// \return kRaiBackendId when built with FLM_ENABLE_RAI, kFlmBackendId otherwise -/// \note A build links one kernel flow or the other, never both, so this is a -/// compile-time fact. It is deliberately *not* derived from -/// utils::platform_id: the backend axis and the platform axis are allowed -/// to disagree, and routing one through the other is what used to make -/// them look like the same thing. -constexpr const char* build_default_backend_id() { -#ifdef FLM_ENABLE_RAI - return kRaiBackendId; -#else - return kFlmBackendId; -#endif -} +/// \brief the entry key naming the kernel flow an entry's artifacts need +/// \note Written onto the entry by model_list, which derives it from the tag, +/// rather than read from model_list.json: the tag is what decides, and +/// it is not part of the entry that reaches AutoModel. An entry without +/// it runs on kFlmBackendId. +inline constexpr const char* kBackendKey = "backend"; + +/// \brief the suffix that marks a model family as ryzenai-corelib's +/// \note The tag is the mechanism: a corelib model is packaged differently +/// from its FastFlowLM namesake, so it is a different model to pull and +/// a different tag to ask for. model_list::backend_for_family is where +/// this is applied; it is spelled out here so the two ids and the rule +/// that picks between them sit together. +inline constexpr const char* kRaiFamilySuffix = "-rai"; /// \brief everything a backend factory needs to build its engine /// \note Assembled by the frontend once the shared model state is initialized, @@ -159,6 +163,13 @@ class BackendRegistry { /// \brief whether a family has a backend with this id bool has(const std::string& family, const std::string& id) const; + /// \brief every backend id this build registers for any family, sorted + /// \note This is what the build links, which is what model_list prunes the + /// catalog against: an entry whose kernel flow is absent is not a + /// model this binary can run, exactly as an entry for the wrong NPU + /// generation is not. + std::vector backend_ids() const; + /// \brief build a backend /// \throws std::runtime_error naming the available ids if it is not registered std::unique_ptr create(const std::string& family, @@ -186,16 +197,18 @@ void register_builtin_backends(BackendRegistry& registry); /// \brief decide which backend to run a model on /// \param family the model family, as in details.family -/// \param fallback the backend id to use when nothing else picks one, normally -/// build_default_backend_id() +/// \param fallback the backend the entry asks for -- its kBackendKey, which +/// model_list derived from the tag, or kFlmBackendId when it has none /// \param requested the --backend value, empty when the flag was not given /// \param source if non-null, receives a human-readable reason for the choice /// \return the resolved backend id /// \throws std::runtime_error naming the registered ids when nothing matches -/// \note Precedence: --backend, then FLM_BACKEND, then `fallback`. The catalog -/// does not name a backend: model_list has already pruned itself to the -/// entries this build's platform can run, and a per-entry backend list -/// would only restate what the build already links. +/// \note Precedence: --backend, then FLM_BACKEND, then `fallback`. Every one +/// of them is a demand, not a preference: an id that this family does +/// not register is an error whoever asked for it. The entry cannot +/// reasonably ask for a flow the build lacks, because model_list has +/// already pruned the entries whose flow is missing -- so a throw here +/// means a genuine mismatch, not a routine packaging gap. /// \note `fallback` arrives as a string so that this header stays free of the /// NPU runtime includes, which is what lets test/model_backend build /// without an XRT toolchain. diff --git a/src/include/model_list.hpp b/src/include/model_list.hpp index 9e115c049..eedc38bc8 100644 --- a/src/include/model_list.hpp +++ b/src/include/model_list.hpp @@ -30,13 +30,26 @@ class model_list { /// \brief constructor /// \param list_path the path to the model list /// \param exe_dir the executable directory for resolving relative paths - /// \param platform the detected NPU generation ("stx" or "aie_next"); the - /// catalog is pruned to the models that generation can run, and - /// each surviving entry has its platform_overrides patch applied + /// \param platform the NPU generation this machine has ("aie2p" or + /// "aie_next"); an entry whose supported_platforms names a + /// different generation is pruned, because these artifacts do + /// not run on this silicon + /// \param backends the kernel flows this build links, as backend ids; + /// an entry whose flow is not among them is pruned too + /// \note Two independent axes. The generation says what the silicon can + /// run; the backend list says which kernels were linked in. Which + /// flow runs an entry is not a third thing to look up: a tag is + /// served by exactly one flow and its name says which, so a + /// family ending in "-rai" is ryzenai-corelib's and anything else + /// is the FastFlowLM kernels'. That is why the two never contend + /// for a tag -- phi4-mini-it and phi4-mini-it-rai are separate + /// entries, with their own artifacts and their own silicon. model_list(std::string& list_path, std::string& exe_dir, - std::string platform = "stx"){ + std::string platform = "aie2p", + std::vector backends = {"flm"}){ this->list_path = list_path; this->platform_ = std::move(platform); + this->backends_ = std::move(backends); std::ifstream config_file(list_path); if (!config_file.is_open()) { std::cerr << "Failed to open config file: " << list_path << std::endl; @@ -52,7 +65,7 @@ class model_list { // Prune before indexing: all_tags must describe what this machine can // actually run, so an unsupported tag fails at validation instead of // failing much later inside the model backend. - this->apply_platform_filter(); + this->apply_support_filter(); // Populate all_tags set for (const auto& [model_type, sizes] : this->config["models"].items()) { @@ -63,15 +76,22 @@ class model_list { } } + // An empty catalog is a real state, not a crash: it is what an + // install looks like when every entry names other silicon or other + // kernels. Say so once, here, where the reason is still known -- + // "Models:" followed by nothing tells the user only that something + // is wrong -- and then let the commands answer for themselves. if (all_tags.empty()) { - header_print_r("ERROR", "No models in " + this->list_path + - " support this NPU (" + this->platform_ + ")"); - exit(1); + header_print_r("ERROR", + "No models in " + this->list_path + + " run on this NPU (" + this->platform_ + + ") with the kernels this build has (" + + this->backend_summary() + ")"); } } /// \brief the NPU generation this catalog was filtered for - /// \return "stx" or "aie_next" + /// \return "aie2p" or "aie_next" const std::string& platform() const { return this->platform_; } /// \brief get the model info @@ -150,7 +170,7 @@ class model_list { // check if size is specified if (new_tag.find(':') == std::string::npos) { const std::string model_type = new_tag; - // A family pruned for this platform (or simply misspelled) has no + // A family this build has no kernels for (or a misspelled one) has no // sizes to pick from. Return the tag untouched so get_model_info // reports it rather than dereferencing a null subset. const auto& models = this->config["models"]; @@ -263,17 +283,55 @@ class model_list { nlohmann::json config; std::string model_root_path; std::string platform_; + /// \brief the backend ids this build links + /// \note Spelled as plain strings rather than reached through + /// flm::backend: AutoModel/model_backend.hpp pulls in causal_lm + /// and the NPU runtime headers, and keeping those out is what + /// lets this class be unit tested without an XRT toolchain. The + /// key name mirrors flm::backend::kBackendKey. + std::vector backends_; + + /// \brief the backend ids, rendered for an error message + std::string backend_summary() const { + std::string joined; + for (const auto& id : this->backends_) { + if (!joined.empty()) joined += ", "; + joined += id; + } + return joined.empty() ? std::string("none") : joined; + } + + /// \brief the kernel flow that runs a tag, read off the tag itself + /// \param model_type the family part of the tag, e.g. "phi4-mini-it-rai" + /// \return "rai" for a family ending in "-rai", "flm" otherwise + /// \note The name is the mechanism, not a label on top of one. A + /// corelib model is packaged differently from its FastFlowLM + /// namesake -- different weights, different quantization -- so it + /// is a different model to pull and a different tag to ask for, + /// and there is nothing left for a catalog key to disambiguate. + static std::string backend_for_family(const std::string& model_type) { + static const std::string suffix = "-rai"; + if (model_type.size() > suffix.size() && + model_type.compare(model_type.size() - suffix.size(), + suffix.size(), suffix) == 0) { + return "rai"; + } + return "flm"; + } - /// \brief whether an entry claims support for the active platform + /// \brief whether an entry claims support for this machine's silicon /// \param entry the size entry /// \param tag the "family:size" tag, used only in error messages /// \return true if the entry runs on this->platform_ - /// \note An entry that says nothing is stx-only. stx is what every - /// model runs on, so the catalog only tags the exceptions: an - /// entry needs "supported_platforms" exactly when it runs on aie_next. + /// \note An entry that says nothing runs on every generation. The + /// shipped catalog names one on every entry, because artifacts + /// are built for a generation and saying so is how a model that + /// cannot run here fails at validation rather than deep inside a + /// kernel; the default is what keeps an older catalog, or a + /// hand-made development tree, readable. bool entry_supports(const nlohmann::json& entry, const std::string& tag) const { const auto supported = entry.find("supported_platforms"); - if (supported == entry.end()) return this->platform_ == "stx"; + if (supported == entry.end()) return true; if (!supported->is_array() || supported->empty()) { throw std::runtime_error( "supported_platforms must be a non-empty array: " + tag); @@ -288,39 +346,38 @@ class model_list { return false; } - /// \brief drop entries this NPU cannot run and flatten the survivors + /// \brief whether this build links a given kernel flow + bool build_has(const std::string& id) const { + for (const auto& have : this->backends_) { + if (have == id) return true; + } + return false; + } + + /// \brief drop entries this build cannot run and settle their backend /// \note After this runs the config has exactly the shape it had before - /// platform support existed, so nothing downstream needs to know - /// which platform was selected. - void apply_platform_filter() { + /// any of this existed, plus one key: "backend", the flow that + /// will run the entry. It is written here rather than read from + /// the file because the tag is what decides it, and the tag is + /// not part of the entry -- by the time AutoModel sees the entry, + /// the name it came from is gone. The generation, by contrast, is + /// settled once the entry is kept, so its key is erased. + void apply_support_filter() { std::vector empty_families; for (auto& [model_type, model_subset] : this->config["models"].items()) { std::vector unsupported_sizes; + const std::string backend = backend_for_family(model_type); for (auto& [size, model_info] : model_subset.items()) { const std::string tag = model_type + ":" + size; - if (!entry_supports(model_info, tag)) { + if (!entry_supports(model_info, tag) || + !this->build_has(backend)) { unsupported_sizes.push_back(size); continue; } - // Take the patch first, then erase the bookkeeping keys, so a - // malformed override can never reintroduce them. - nlohmann::json patch = nlohmann::json::object(); - const auto overrides = model_info.find("platform_overrides"); - if (overrides != model_info.end()) { - if (!overrides->is_object()) { - throw std::runtime_error( - "platform_overrides must be an object: " + tag); - } - const auto match = overrides->find(this->platform_); - if (match != overrides->end()) patch = *match; - } - model_info.erase("platform_overrides"); model_info.erase("supported_platforms"); - // merge_patch replaces arrays wholesale, which is what "files" - // needs, and a null value deletes the key (e.g. "ms_url"). - if (!patch.empty()) model_info.merge_patch(patch); + model_info["backend"] = backend; } for (const auto& size : unsupported_sizes) model_subset.erase(size); @@ -335,7 +392,7 @@ class model_list { /// \brief the entry to fall back on when a tag cannot be resolved /// \return the fallback tag and its info /// \note llama3.2:1b is the historical default, but it is pruned on - /// platforms that cannot run it, so fall back to whatever survived. + /// silicon that cannot run it, so fall back to whatever survived. std::pair fallback_model() const { const auto& models = this->config["models"]; if (models.contains("llama3.2") && @@ -348,7 +405,8 @@ class model_list { } } throw std::runtime_error("No models available for NPU platform " + - this->platform_); + this->platform_ + " with backends " + + this->backend_summary()); } }; diff --git a/src/include/utils/npu_platform.hpp b/src/include/utils/npu_platform.hpp index b00acdcad..5b1730a02 100644 --- a/src/include/utils/npu_platform.hpp +++ b/src/include/utils/npu_platform.hpp @@ -1,13 +1,11 @@ /// \file npu_platform.hpp -/// \brief the NPU generation this build targets (stx vs aie_next) -/// \note A build carries engines for exactly one generation: the two share -/// none, and FLM_ENABLE_RAI picks which at compile time, so the -/// generation is a build-time fact rather than something to go and ask -/// the hardware. The catalog in model_list.json declares which -/// generations each model supports, and model_list prunes itself to the -/// one this build was made for. Which *kernels* serve a generation is a -/// separate question with its own vocabulary -- see kFlmBackendId and -/// kRaiBackendId in AutoModel/model_backend.hpp. +/// \brief the NPU generation a build runs on (aie2p vs aie_next) +/// \note The generation is a property of the silicon, not of the build: it +/// says which model artifacts this machine can run at all, and the +/// catalog in model_list.json names one on every entry. Which *kernels* +/// execute them is a second, independent axis with its own vocabulary -- +/// see kFlmBackendId and kRaiBackendId in AutoModel/model_backend.hpp -- +/// and neither one implies the other. /// \note aie_next names silicon that has not been announced. When it ships, /// this enumerator, the string platform_id() returns, and the catalog /// key in model_list.json are the whole of the rename; nothing on disk @@ -22,33 +20,42 @@ namespace utils { /// \brief NPU generation an FLM build targets enum class npu_platform { - stx, ///< Strix / Krackan Point + aie2p, ///< the AIE2P generation: Strix / Strix Halo / Krackan Point aie_next ///< the next NPU generation, not yet announced }; /// \brief the catalog id for a platform, as written in model_list.json /// \param platform the platform -/// \return "stx" or "aie_next" +/// \return "aie2p" or "aie_next" constexpr std::string_view platform_id(npu_platform platform) { return platform == npu_platform::aie_next ? std::string_view("aie_next") - : std::string_view("stx"); + : std::string_view("aie2p"); } -/// \brief the platform every entry is assumed to support when it says nothing -constexpr npu_platform default_npu_platform() { return npu_platform::stx; } - -/// \brief the NPU generation this build has engines for -/// \return aie_next when built with FLM_ENABLE_RAI, stx otherwise -/// \note There is no binary that carries both engines, so this is the whole of -/// platform selection: no probe, and nothing for a user to configure -/// beyond choosing the build that matches their machine. -constexpr npu_platform build_npu_platform() { -#ifdef FLM_ENABLE_RAI - return npu_platform::aie_next; -#else - return npu_platform::stx; -#endif -} +/// \brief the generation assumed when nothing has read the device +/// \note Until the probe in get_device() is real this is not a default but the +/// whole answer, so it decides what every install can see. It is +/// aie_next while the corelib path is what is being brought up: that is +/// the generation corelib's kernels are built for, and an install that +/// claimed aie2p could not offer them at all. +/// \note The cost is exact and intended: every entry in model_list.json except +/// the corelib one names aie2p, so a build that does not link corelib +/// offers nothing here. That is not a broken install, it is an install +/// for silicon none of its models were built for, and model_list says so +/// in one line. Set this back to aie2p to get those 42 models back. +constexpr npu_platform default_npu_platform() { return npu_platform::aie_next; } + +/// \brief ask the machine which NPU generation it has +/// \return the generation of the NPU in this host +/// \note A stand-in, and deliberately the dumbest one that can be written: the +/// real probe reads the device and is not part of this tree, so until it +/// lands this returns default_npu_platform() and consults nothing at all +/// -- not the environment, and not which kernels were linked. A build +/// that links corelib is still a build on whatever silicon it is running +/// on, and an install that offers a model it cannot run is worse than one +/// that offers nothing. The call sites need no change when the probe +/// arrives: they already treat the generation as a run-time answer. +npu_platform get_device(); /// \brief parse a catalog platform id /// \param text the id, e.g. "aie_next" diff --git a/src/include/utils/utils.hpp b/src/include/utils/utils.hpp index aaa8e66e0..6766c9cbf 100644 --- a/src/include/utils/utils.hpp +++ b/src/include/utils/utils.hpp @@ -9,14 +9,10 @@ #include "typedef.hpp" #include "buffer.hpp" #include "debug_utils.hpp" -#include "device_runtime.hpp" -#include "nlohmann/json.hpp" #include #include #include #include -#include -#include #ifdef _WIN32 #include #include diff --git a/src/model_list.json b/src/model_list.json index 92be08133..9aefd6052 100644 --- a/src/model_list.json +++ b/src/model_list.json @@ -3,6 +3,7 @@ "models": { "hy-mt2": { "1.8b": { + "supported_platforms": ["aie2p"], "name": "Hy-MT2-1.8B-NPU2", "url": "https://huggingface.co/FastFlowLM/Hy-MT2-1.8B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Hy-MT2-1.8B-NPU2/tree/main", @@ -37,6 +38,7 @@ }, "nanbeige4.1": { "3b": { + "supported_platforms": ["aie2p"], "name": "Nanbeige4.1-3B-NPU2", "url": "https://huggingface.co/FastFlowLM/Nanbeige4.1-3B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Nanbeige4.1-3B-NPU2/tree/main", @@ -58,7 +60,7 @@ "parameter_size": "3B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "reasoning" ], "footprint": 3.1 @@ -66,6 +68,7 @@ }, "qwen3vl-it": { "4b": { + "supported_platforms": ["aie2p"], "name": "Qwen3-VL-4B-Instruct-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3-VL-4B-Instruct-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3-VL-4B-Instruct-NPU2/tree/main", @@ -98,6 +101,7 @@ }, "qwen3vl-flash": { "4b": { + "supported_platforms": ["aie2p"], "name": "Qwen3-VL-4B-Instruct-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3-VL-4B-Instruct-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3-VL-4B-Instruct-NPU2/tree/main", @@ -130,6 +134,7 @@ }, "gemma4-it": { "e2b": { + "supported_platforms": ["aie2p"], "name": "Gemma4-E2B-IT-NPU2", "url": "https://huggingface.co/FastFlowLM/Gemma4-E2B-IT-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Gemma4-E2B-IT-NPU2/tree/main", @@ -166,6 +171,7 @@ "footprint": 6.0 }, "e4b": { + "supported_platforms": ["aie2p"], "name": "Gemma4-E4B-IT-NPU2", "url": "https://huggingface.co/FastFlowLM/Gemma4-E4B-IT-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Gemma4-E4B-IT-NPU2/tree/main", @@ -202,6 +208,7 @@ "footprint": 9.1 }, "12b": { + "supported_platforms": ["aie2p"], "name": "Gemma4-12B-IT-NPU2", "url": "https://huggingface.co/FastFlowLM/Gemma4-12B-IT-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Gemma4-12B-IT-NPU2/tree/main", @@ -239,6 +246,7 @@ }, "gemma4e-flash": { "e2b": { + "supported_platforms": ["aie2p"], "name": "Gemma4-E2B-IT-NPU2", "url": "https://huggingface.co/FastFlowLM/Gemma4-E2B-IT-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Gemma4-E2B-IT-NPU2/tree/main", @@ -275,6 +283,7 @@ "footprint": 6.0 }, "e4b": { + "supported_platforms": ["aie2p"], "name": "Gemma4-E4B-IT-NPU2", "url": "https://huggingface.co/FastFlowLM/Gemma4-E4B-IT-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Gemma4-E4B-IT-NPU2/tree/main", @@ -313,6 +322,7 @@ }, "qwen3.5": { "0.8b": { + "supported_platforms": ["aie2p"], "name": "Qwen3.5-0.8B-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3.5-0.8B-NPU2/resolve/flm_q4k_high_precision", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3.5-0.8B-NPU2/tree/flm_q4k_high_precision", @@ -344,6 +354,7 @@ "footprint": 1.3 }, "2b": { + "supported_platforms": ["aie2p"], "name": "Qwen3.5-2B-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3.5-2B-NPU2/resolve/flm_q4k_high_precision", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3.5-2B-NPU2/tree/flm_q4k_high_precision", @@ -376,6 +387,7 @@ "footprint": 3.1 }, "4b": { + "supported_platforms": ["aie2p"], "name": "Qwen3.5-4B-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3.5-4B-NPU2/resolve/flm_q4k_high_precision", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3.5-4B-NPU2/tree/flm_q4k_high_precision", @@ -407,7 +419,8 @@ ], "footprint": 5.0 }, - "9b": { + "9b":{ + "supported_platforms": ["aie2p"], "name": "Qwen3.5-9B-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3.5-9B-NPU2/resolve/flm_q4k_high_precision", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3.5-9B-NPU2/tree/flm_q4k_high_precision", @@ -440,8 +453,10 @@ "footprint": 8.7 } }, + "qwen3.6-moe": { "35b-a3b": { + "supported_platforms": ["aie2p"], "name": "Qwen3.6-35B-A3B-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3.6-35B-A3B-NPU2/resolve/flm_q4k_high_precision", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3.6-35B-A3B-NPU2/tree/flm_q4k_high_precision", @@ -476,6 +491,7 @@ }, "lfm2": { "1.2b": { + "supported_platforms": ["aie2p"], "name": "LFM2-1.2B-NPU2", "url": "https://huggingface.co/FastFlowLM/LFM2-1.2B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/LFM2-1.2B-NPU2/tree/main", @@ -502,6 +518,7 @@ "footprint": 0.96 }, "2.6b": { + "supported_platforms": ["aie2p"], "name": "LFM2-2.6B-NPU2", "url": "https://huggingface.co/FastFlowLM/LFM2-2.6B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/LFM2-2.6B-NPU2/tree/main", @@ -530,6 +547,7 @@ }, "lfm2-trans": { "2.6b": { + "supported_platforms": ["aie2p"], "name": "LFM2-2.6B-Transcript-NPU2", "url": "https://huggingface.co/FastFlowLM/LFM2-2.6B-Transcript-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/LFM2-2.6B-Transcript-NPU2/tree/main", @@ -558,6 +576,7 @@ }, "lfm2.5-it": { "1.2b": { + "supported_platforms": ["aie2p"], "name": "LFM2.5-1.2B-NPU2", "url": "https://huggingface.co/FastFlowLM/LFM2.5-1.2B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/LFM2.5-1.2B-NPU2/tree/main", @@ -586,6 +605,7 @@ }, "lfm2.5-tk": { "1.2b": { + "supported_platforms": ["aie2p"], "name": "LFM2.5-1.2B-Thinking-NPU2", "url": "https://huggingface.co/FastFlowLM/LFM2.5-1.2B-Thinking-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/LFM2.5-1.2B-Thinking-NPU2/tree/main", @@ -610,7 +630,7 @@ "tokenizer_config.json", "chat_template.jinja" ], - "label": [ + "label":[ "reasoning" ], "footprint": 0.96 @@ -618,6 +638,7 @@ }, "phi4-mini-it": { "4b": { + "supported_platforms": ["aie2p"], "name": "Phi4-mini-Instruct-NPU2", "url": "https://huggingface.co/FastFlowLM/Phi4-mini-Instruct-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Phi4-mini-Instruct-NPU2/tree/main", @@ -640,51 +661,53 @@ "tokenizer.json", "tokenizer_config.json" ], - "footprint": 3.4, - "supported_platforms": [ - "stx", - "aie_next" - ], - "platform_overrides": { - "aie_next": { - "name": "phi4-mini-it-rai", - "url": "https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF/resolve/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", - "file_url": "https://huggingface.co/api/models/unsloth/Phi-4-mini-instruct-GGUF/tree/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", - "size": 4100140571, - "default_context_length": 4096, - "details": { - "quantization_level": "Q8_0 -> group-64" - }, - "flm_min_version": "1.0.3", - "files": [ - "Phi-4-mini-instruct.Q8_0.gguf", - "tokenizer.json", - "tokenizer_config.json", - "config.json" - ], - "file_sources": { - "tokenizer.json": { - "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", - "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" - }, - "tokenizer_config.json": { - "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", - "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" - }, - "config.json": { - "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", - "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" - } - }, - "footprint": 4.1, - "ms_url": null, - "model_info_key": "phi4-mini-it-rai:4b" + "footprint": 3.4 + } + }, + "phi4-mini-it-rai": { + "4b": { + "supported_platforms": ["aie_next"], + "name": "phi4-mini-it-rai", + "url": "https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF/resolve/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", + "file_url": "https://huggingface.co/api/models/unsloth/Phi-4-mini-instruct-GGUF/tree/78eb92a46fc37e6b524df991ed9aca9bc6aa7b80", + "size": 4100140571, + "default_context_length": 4096, + "max_prefill_len": 4096, + "details": { + "family": "phi4", + "think": false, + "think_toggleable": false, + "parameter_size": "4B", + "quantization_level": "Q8_0 -> group-64" + }, + "flm_min_version": "1.0.3", + "vlm": false, + "files": [ + "Phi-4-mini-instruct.Q8_0.gguf", + "tokenizer.json", + "tokenizer_config.json", + "config.json" + ], + "file_sources": { + "tokenizer.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + }, + "tokenizer_config.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" + }, + "config.json": { + "url": "https://huggingface.co/microsoft/Phi-4-mini-instruct", + "revision": "cfbefacb99257ffa30c83adab238a50856ac3083" } - } + }, + "footprint": 4.1 } }, "embed-gemma": { "300m": { + "supported_platforms": ["aie2p"], "name": "Embedding-Gemma-300M-NPU2", "url": "https://huggingface.co/FastFlowLM/Embedding-Gemma-300M-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Embedding-Gemma-300M-NPU2/tree/main", @@ -705,7 +728,7 @@ "parameter_size": "300M", "quantization_level": "none" }, - "label": [ + "label":[ "embeddings" ], "footprint": 0.62 @@ -713,6 +736,7 @@ }, "whisper-v3": { "turbo": { + "supported_platforms": ["aie2p"], "name": "Whisper-V3-Turbo-NPU2", "url": "https://huggingface.co/FastFlowLM/Whisper-V3-Turbo-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Whisper-V3-Turbo-NPU2/tree/main", @@ -734,7 +758,7 @@ "parameter_size": "1B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "audio", "realtime-transcription", "transcription" @@ -744,6 +768,7 @@ }, "gemma3": { "1b": { + "supported_platforms": ["aie2p"], "name": "Gemma3-1B-NPU2", "url": "https://huggingface.co/FastFlowLM/Gemma3-1B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Gemma3-1B-NPU2/tree/main", @@ -769,6 +794,7 @@ "footprint": 1.2 }, "4b": { + "supported_platforms": ["aie2p"], "name": "Gemma3-4B-NPU2", "url": "https://huggingface.co/FastFlowLM/Gemma3-4B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Gemma3-4B-NPU2/tree/main", @@ -793,7 +819,7 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "vision" ], "footprint": 4.5 @@ -801,6 +827,7 @@ }, "translategemma": { "4b": { + "supported_platforms": ["aie2p"], "name": "Translategemma-4B-Instruct-NPU2", "url": "https://huggingface.co/FastFlowLM/Translategemma-4B-Instruct-NPU2/resolve/main", "file_url": "https://huggingface.co/api/models/FastFlowLM/Translategemma-4B-Instruct-NPU2/tree/main", @@ -825,7 +852,7 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "vision" ], "footprint": 4.5 @@ -833,6 +860,7 @@ }, "medgemma": { "4b": { + "supported_platforms": ["aie2p"], "name": "Medgemma-4B-NPU2", "url": "https://huggingface.co/FastFlowLM/medgemma-4b-it-NPU2/resolve/main", "file_url": "https://huggingface.co/api/models/FastFlowLM/medgemma-4b-it-NPU2/tree/main", @@ -857,7 +885,7 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "vision" ], "footprint": 4.5 @@ -865,6 +893,7 @@ }, "medgemma1.5": { "4b": { + "supported_platforms": ["aie2p"], "name": "Medgemma-1.5-4B-NPU2", "url": "https://huggingface.co/FastFlowLM/medgemma-1.5-4b-it-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/medgemma-1.5-4b-it-NPU2/tree/main", @@ -889,7 +918,7 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "vision" ], "footprint": 4.5 @@ -897,6 +926,7 @@ }, "llama3.2": { "1b": { + "supported_platforms": ["aie2p"], "name": "Llama-3.2-1B-NPU2", "url": "https://huggingface.co/FastFlowLM/Llama-3.2-1B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Llama-3.2-1B-NPU2/tree/main", @@ -921,6 +951,7 @@ "footprint": 1.3 }, "3b": { + "supported_platforms": ["aie2p"], "name": "Llama-3.2-3B-NPU2", "url": "https://huggingface.co/FastFlowLM/Llama-3.2-3B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Llama-3.2-3B-NPU2/tree/main", @@ -947,6 +978,7 @@ }, "llama3.1": { "8b": { + "supported_platforms": ["aie2p"], "name": "Llama-3.1-8B-NPU2", "url": "https://huggingface.co/FastFlowLM/Llama-3.1-8B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Llama-3.1-8B-NPU2/tree/main", @@ -973,6 +1005,7 @@ }, "deepseek-r1": { "8b": { + "supported_platforms": ["aie2p"], "name": "Deepseek-R1-Distill-Llama-8B-NPU2", "url": "https://huggingface.co/FastFlowLM/Deepseek-R1-Distill-Llama-8B-NPU2/resolve/main", "file_url": "https://huggingface.co/api/models/FastFlowLM/Deepseek-R1-Distill-Llama-8B-NPU2/tree/main", @@ -994,7 +1027,7 @@ "parameter_size": "8B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "reasoning" ], "footprint": 5.4 @@ -1002,6 +1035,7 @@ }, "deepseek-r1-0528": { "8b": { + "supported_platforms": ["aie2p"], "name": "DeepSeek-R1-0528-Qwen3-8B-NPU2", "url": "https://huggingface.co/FastFlowLM/DeepSeek-R1-0528-Qwen3-8B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/DeepSeek-R1-0528-Qwen3-8B-NPU2/tree/main", @@ -1024,7 +1058,7 @@ "parameter_size": "8B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "reasoning" ], "footprint": 5.6 @@ -1032,6 +1066,7 @@ }, "qwen3": { "0.6b": { + "supported_platforms": ["aie2p"], "name": "Qwen3-0.6B-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3-0.6B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3-0.6B-NPU2/tree/main", @@ -1054,12 +1089,13 @@ "parameter_size": "0.6B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "reasoning" ], "footprint": 0.66 }, "1.7b": { + "supported_platforms": ["aie2p"], "name": "Qwen3-1.7B-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3-1.7B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3-1.7B-NPU2/tree/main", @@ -1082,12 +1118,13 @@ "parameter_size": "1.7B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "reasoning" ], "footprint": 1.6 }, "4b": { + "supported_platforms": ["aie2p"], "name": "Qwen3-4B-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3-4B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3-4B-NPU2/tree/main", @@ -1110,13 +1147,14 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "reasoning", "tool-calling" ], "footprint": 3.1 }, "8b": { + "supported_platforms": ["aie2p"], "name": "Qwen3-8B-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3-8B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3-8B-NPU2/tree/main", @@ -1139,7 +1177,7 @@ "parameter_size": "8B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "reasoning", "tool-calling" ], @@ -1148,6 +1186,7 @@ }, "qwen3-tk": { "4b": { + "supported_platforms": ["aie2p"], "name": "Qwen3-4B-Thinking-2507-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3-4B-Thinking-2507-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3-4B-Thinking-2507-NPU2/tree/main", @@ -1170,7 +1209,7 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "reasoning", "tool-calling" ], @@ -1179,6 +1218,7 @@ }, "qwen3-it": { "4b": { + "supported_platforms": ["aie2p"], "name": "Qwen3-4B-Instruct-2507-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen3-4B-Instruct-2507-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen3-4B-Instruct-2507-NPU2/tree/main", @@ -1201,7 +1241,7 @@ "parameter_size": "4B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "tool-calling" ], "footprint": 3.1 @@ -1209,6 +1249,7 @@ }, "gpt-oss": { "20b": { + "supported_platforms": ["aie2p"], "name": "GPT-OSS-20B-NPU2", "url": "https://huggingface.co/FastFlowLM/GPT-OSS-20B-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/GPT-OSS-20B-NPU2/tree/main", @@ -1231,7 +1272,7 @@ "parameter_size": "20B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "reasoning" ], "footprint": 14.0 @@ -1239,6 +1280,7 @@ }, "gpt-oss-sg": { "20b": { + "supported_platforms": ["aie2p"], "name": "GPT-OSS-Safeguard-20b-NPU2", "url": "https://huggingface.co/FastFlowLM/GPT-OSS-Safeguard-20b-NPU2", "file_url": "https://huggingface.co/api/models/FastFlowLM/GPT-OSS-Safeguard-20b-NPU2/tree/main", @@ -1261,7 +1303,7 @@ "parameter_size": "20B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "reasoning" ], "footprint": 14.0 @@ -1269,6 +1311,7 @@ }, "qwen2.5-it": { "3b": { + "supported_platforms": ["aie2p"], "name": "Qwen2.5-3B-Instruct-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen2.5-3B-Instruct-NPU2/resolve/main", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen2.5-3B-Instruct-NPU2/tree/main", @@ -1296,6 +1339,7 @@ }, "qwen2.5vl-it": { "3b": { + "supported_platforms": ["aie2p"], "name": "Qwen2.5-VL-3B-Instruct-NPU2", "url": "https://huggingface.co/FastFlowLM/Qwen2.5-VL-3B-Instruct-NPU2/resolve/main", "file_url": "https://huggingface.co/api/models/FastFlowLM/Qwen2.5-VL-3B-Instruct-NPU2/tree/main", @@ -1320,7 +1364,7 @@ "parameter_size": "3B", "quantization_level": "Q4_1" }, - "label": [ + "label":[ "vision" ], "footprint": 3.8 diff --git a/src/src/main.cpp b/src/src/main.cpp index 061e5a05c..e32fca6ab 100644 --- a/src/src/main.cpp +++ b/src/src/main.cpp @@ -8,6 +8,7 @@ #include "runner.hpp" #include "server.hpp" #include "model_list.hpp" +#include "AutoModel/model_backend.hpp" #include "model_downloader.hpp" #include "update.hpp" #include "utils/utils.hpp" @@ -228,9 +229,9 @@ static bool sanity_check_npu_stack(bool quiet, bool json_output = false) { {"ready", true} }; validation_json["platform"] = "linux"; - // The same build-time generation the catalog was filtered for in main(). + // The same generation the catalog was filtered for in main(). validation_json["npu_platform"] = - std::string(utils::platform_id(utils::build_npu_platform())); + std::string(utils::platform_id(utils::get_device())); // Check kernel version struct utsname u_name; if (uname(&u_name) != 0) { @@ -411,7 +412,7 @@ static bool sanity_check_npu_stack(bool quiet, bool json_output = false) { {"amd_device_found", true}, {"npu_driver_ok", true}, {"npu_platform", - std::string(utils::platform_id(utils::build_npu_platform()))}, + std::string(utils::platform_id(utils::get_device()))}, {"ready", true} }; std::string npu_arch = identify_npu_arch(); @@ -627,13 +628,15 @@ int main(int argc, char* argv[]) { flm_rt::device* npu_device = open_npu_device(&npu_open_error); #endif - // Which generation this binary is for is decided by FLM_ENABLE_RAI at - // build time: the two generations share no engine, so a build has one of - // them and there is nothing to detect. - constexpr utils::npu_platform platform = utils::build_npu_platform(); + // Two things decide what this binary can run, and they are independent. + // The generation is the machine's, answered by the device itself, and says + // which artifacts run here at all; the backend ids are the kernel flows + // that were linked in, and a tag names its own flow. + const utils::npu_platform platform = utils::get_device(); - model_list availble_models(config_path, models_dir, - std::string(utils::platform_id(platform))); + model_list availble_models( + config_path, models_dir, std::string(utils::platform_id(platform)), + flm::backend::BackendRegistry::instance().backend_ids()); const bool print_status = !parsed_args.json_output && !parsed_args.sub_process_mode; const bool needs_npu = @@ -659,20 +662,6 @@ int main(int argc, char* argv[]) { : ": " + npu_open_error)); } - // The rai build of phi4-mini-it installs under its own directory name, so - // on any other platform that directory is no longer reachable by a tag and - // `flm remove` cannot clean it up. Point it out; never delete it. - if (print_status && platform != utils::npu_platform::aie_next) { - const std::filesystem::path stale_dir = - std::filesystem::path(availble_models.get_model_root_path()) / "phi4-mini-it-rai"; - std::error_code stale_ec; - if (std::filesystem::exists(stale_dir, stale_ec)) { - header_print("FLM", "Note: " << stale_dir.string() - << " is a rai-only model and is unused on this NPU; " - "delete it manually to reclaim the space."); - } - } - // Extract parsed values bool got_power_mode = (parsed_args.power_mode != "performance"); // Check if user explicitly set power mode bool stable_stack = false; diff --git a/src/test/model_backend/test_model_backend.cpp b/src/test/model_backend/test_model_backend.cpp index 5ea890aff..463b247b4 100644 --- a/src/test/model_backend/test_model_backend.cpp +++ b/src/test/model_backend/test_model_backend.cpp @@ -13,6 +13,7 @@ #include #include +#include #include using flm::backend::BackendContext; @@ -173,20 +174,20 @@ void test_unknown_id_names_what_exists() { RequireContains(empty, "(none)"); } -void test_the_build_default_is_the_default() { +void test_the_catalog_entry_is_the_default() { auto& registry = BackendRegistry::instance(); registry.replace_backend("phi4", kFlmBackendId, StubFactory("flm")); registry.replace_backend("phi4", kRaiBackendId, StubFactory("rai")); std::string source; - // Nothing overrides it, so the build's own default decides. It is passed - // in rather than detected: which kernel provider a binary links is fixed - // at compile time, and this function has no business asking hardware. + // Nothing overrides it, so the entry decides. Both flows are registered + // for this family at once, which is the point: they are two ways to run + // phi4 on one machine, not two machines. TEST_REQUIRE(resolve_backend_id("phi4", "flm", "", &source) == kFlmBackendId); - TEST_REQUIRE(source == "build default"); + TEST_REQUIRE(source == "model catalog"); TEST_REQUIRE(resolve_backend_id("phi4", "rai", "", &source) == kRaiBackendId); - TEST_REQUIRE(source == "build default"); + TEST_REQUIRE(source == "model catalog"); } void test_resolution_precedence() { @@ -196,7 +197,7 @@ void test_resolution_precedence() { std::string source; { - // FLM_BACKEND beats the build default. + // FLM_BACKEND beats what the entry asked for. ScopedBackendEnv env(kRaiBackendId); TEST_REQUIRE(resolve_backend_id("phi4", "flm", "", &source) == kRaiBackendId); TEST_REQUIRE(source == "FLM_BACKEND"); @@ -210,7 +211,39 @@ void test_resolution_precedence() { // An empty FLM_BACKEND is the same as an unset one. ScopedBackendEnv empty(""); TEST_REQUIRE(resolve_backend_id("phi4", "rai", "", &source) == kRaiBackendId); - TEST_REQUIRE(source == "build default"); + TEST_REQUIRE(source == "model catalog"); +} + +void test_adding_rai_leaves_the_flm_families_alone() { + auto& registry = BackendRegistry::instance(); + registry.replace_backend("llama3", kFlmBackendId, StubFactory("flm")); + registry.replace_backend("phi4", kFlmBackendId, StubFactory("flm")); + registry.replace_backend("phi4", kRaiBackendId, StubFactory("rai")); + + // This is the shape of a corelib build: rai for the one family that has a + // corelib engine, flm for that family too, and flm alone everywhere else. + // Linking rai must not disturb any of the families it says nothing about. + std::string source; + TEST_REQUIRE(resolve_backend_id("llama3", kFlmBackendId, "", &source) == + kFlmBackendId); + TEST_REQUIRE(source == "model catalog"); + TEST_REQUIRE(resolve_backend_id("phi4", kRaiBackendId, "", &source) == + kRaiBackendId); + TEST_REQUIRE(source == "model catalog"); + + // rai is registered, but not for llama3, and nothing quietly substitutes + // another flow: asking for one backend and silently getting a different + // one would be worse than an error. The catalog cannot ask for this in + // practice -- model_list prunes an entry whose flow is missing -- so a + // throw here means a real mismatch rather than a packaging gap. + RequireThrows([&] { resolve_backend_id("llama3", kRaiBackendId); }); + RequireThrows( + [&] { resolve_backend_id("llama3", kFlmBackendId, kRaiBackendId); }); + + // Every backend the build links, whichever family registered it. + const auto ids = registry.backend_ids(); + TEST_REQUIRE(std::find(ids.begin(), ids.end(), kFlmBackendId) != ids.end()); + TEST_REQUIRE(std::find(ids.begin(), ids.end(), kRaiBackendId) != ids.end()); } void test_resolution_rejects_with_a_readable_message() { @@ -222,13 +255,13 @@ void test_resolution_rejects_with_a_readable_message() { const std::string not_built = RequireThrows( [&] { resolve_backend_id("llama3", "flm", kRaiBackendId); }); RequireContains(not_built, "--backend"); - RequireContains(not_built, "not compiled into this build"); + RequireContains(not_built, "is not available for model family"); RequireContains(not_built, "flm"); // Same for a provider nobody has an engine for yet. const std::string unknown_provider = RequireThrows([&] { resolve_backend_id("llama3", "gpu"); }); - RequireContains(unknown_provider, "build default"); + RequireContains(unknown_provider, "model catalog"); RequireContains(unknown_provider, "gpu"); // The env var gets named in the message too, so the user can find it. @@ -252,8 +285,10 @@ int main() { RunTest(test_duplicate_registration_is_rejected, "duplicate registration is rejected"); RunTest(test_replace_backend_is_the_test_seam, "replace_backend is the test seam"); RunTest(test_unknown_id_names_what_exists, "unknown id names what exists"); - RunTest(test_the_build_default_is_the_default, "the build default is the default"); + RunTest(test_the_catalog_entry_is_the_default, "the catalog entry is the default"); RunTest(test_resolution_precedence, "resolution precedence"); + RunTest(test_adding_rai_leaves_the_flm_families_alone, + "adding rai leaves the flm families alone"); RunTest(test_resolution_rejects_with_a_readable_message, "resolution rejects with a readable message"); std::cout << "All model backend tests passed\n"; diff --git a/src/test/model_list_platform/test_model_list_platform.cpp b/src/test/model_list_platform/test_model_list_platform.cpp index 99039e561..8f3129720 100644 --- a/src/test/model_list_platform/test_model_list_platform.cpp +++ b/src/test/model_list_platform/test_model_list_platform.cpp @@ -1,211 +1,337 @@ /// \file test_model_list_platform.cpp -/// \brief Platform filtering / override merging in model_list, plus the -/// npu_platform helpers and a sweep of the shipped catalog. +/// \brief Platform and backend filtering in model_list, plus the npu_platform +/// helpers and a sweep of the shipped catalog. /// \note Deliberately free of NPU hardware: everything here is catalog logic, -/// so it builds and runs on Linux CI where the phi4_rai suite -/// cannot. +/// so it builds and runs on Linux CI where the phi4_rai suite cannot. #include "model_list.hpp" #include "utils/npu_platform.hpp" #include "../phi4_rai/test_support.hpp" -#include #include +#include +#include #include +#include +#include namespace { constexpr const char* kCatalogPath = FLM_TEST_MODEL_LIST_PATH; constexpr const char* kPhiTag = "phi4-mini-it:4b"; +constexpr const char* kRaiTag = "phi4-mini-it-rai:4b"; -/// \brief build a model_list over the shipped catalog for one platform -model_list open_catalog(const std::string& platform) { +/// \brief every generation the build knows about, for the catalog sweep +const std::set kAllPlatforms = { + std::string(utils::platform_id(utils::npu_platform::aie2p)), + std::string(utils::platform_id(utils::npu_platform::aie_next))}; + +/// \brief open the shipped catalog as one machine and build read it +/// \param platform the generation the device reports +/// \param backends the kernel flows this build is to be taken as linking +model_list open_catalog(const std::string& platform, + std::vector backends) { std::string path = kCatalogPath; std::string exe_dir = "."; - return model_list(path, exe_dir, platform); + return model_list(path, exe_dir, platform, std::move(backends)); } -nlohmann::json read_catalog() { - std::ifstream stream(kCatalogPath); +/// \brief the catalog as a stock build on shipping silicon reads it +model_list open_stock() { return open_catalog("aie2p", {"flm"}); } + +/// \brief the catalog as a corelib build on the next generation reads it +model_list open_rai() { return open_catalog("aie_next", {"flm", "rai"}); } + +nlohmann::json read_json(const char* path) { + std::ifstream stream(path); TEST_REQUIRE(stream.is_open()); return nlohmann::json::parse(stream); } -void test_stx_entry_is_unchanged() { - auto models = open_catalog("stx"); +/// \brief write a catalog to a temp file and hand back its path +std::filesystem::path write_catalog(const char* name, const nlohmann::json& catalog) { + const auto path = std::filesystem::temp_directory_path() / name; + std::ofstream out(path); + out << catalog.dump(2); + return path; +} + +void test_stock_build_offers_phi4_with_its_own_artifacts() { + // The regression this design exists to prevent. A stock install has one + // phi4 -- the FastFlowLM one -- and nothing about a corelib model may + // prune it. + auto models = open_stock(); + TEST_REQUIRE(models.is_model_supported(kPhiTag)); + TEST_REQUIRE(models.rectify_model_tag("phi4-mini-it") == kPhiTag); + const auto [tag, info] = models.get_model_info(kPhiTag); TEST_REQUIRE(tag == kPhiTag); TEST_REQUIRE(info.at("name") == "Phi4-mini-Instruct-NPU2"); - TEST_REQUIRE(info.contains("ms_url")); - TEST_REQUIRE(!info.contains("file_sources")); TEST_REQUIRE(info.at("default_context_length") == 32768); TEST_REQUIRE(info.at("flm_min_version") == "0.9.25"); - // The stx catalog is the full catalog, and the aie_next-only entry is gone. + TEST_REQUIRE(info.contains("ms_url")); + TEST_REQUIRE(!info.contains("file_sources")); + // The tag names no flow, so it is the FastFlowLM one. + TEST_REQUIRE(info.at("backend") == "flm"); + // Bookkeeping never reaches the caller. + TEST_REQUIRE(!info.contains("supported_backend")); + TEST_REQUIRE(models.is_model_supported("llama3.2:1b")); - TEST_REQUIRE(!models.is_model_supported("phi4-mini-it-rai:4b")); + // A build with no corelib kernels cannot offer a corelib model. + TEST_REQUIRE(!models.is_model_supported(kRaiTag)); TEST_REQUIRE(!models.is_model_supported("phi4-mini-it-rai")); } -void test_aie_next_entry_is_merged() { - auto models = open_catalog("aie_next"); - TEST_REQUIRE(models.all_tags.size() == 2); - TEST_REQUIRE(models.is_model_supported("phi4-mini-it")); - TEST_REQUIRE(models.is_model_supported(kPhiTag)); +void test_a_corelib_build_offers_the_corelib_phi4() { + // The corelib entry is described by its own artifacts -- a Q8_0 GGUF, not + // the NPU2/Q4NX package -- under its own tag, and it is offered by a build + // that has corelib's kernels, on the silicon they were built for. + auto corelib = open_rai(); + TEST_REQUIRE(corelib.is_model_supported(kRaiTag)); - const auto [tag, info] = models.get_model_info(kPhiTag); - TEST_REQUIRE(tag == kPhiTag); - // The override wins where it speaks... + const auto [tag, info] = corelib.get_model_info(kRaiTag); + TEST_REQUIRE(tag == kRaiTag); + TEST_REQUIRE(corelib.rectify_model_tag("phi4-mini-it-rai") == kRaiTag); TEST_REQUIRE(info.at("name") == "phi4-mini-it-rai"); TEST_REQUIRE(info.at("default_context_length") == 4096); TEST_REQUIRE(info.at("flm_min_version") == "1.0.3"); - TEST_REQUIRE(info.at("model_info_key") == "phi4-mini-it-rai:4b"); TEST_REQUIRE(info.at("files").size() == 4); TEST_REQUIRE(info.at("file_sources").size() == 3); - // ...a null in the patch deletes the key... + // Its own package, pulled from its own upstream: no ModelScope mirror, and + // the tag is the model_info.json key, so nothing has to redirect it. TEST_REQUIRE(!info.contains("ms_url")); - // ...and the rest of details survives the recursive merge. + TEST_REQUIRE(!info.contains("model_info_key")); + // Same engine family, other kernels. TEST_REQUIRE(info.at("details").at("family") == "phi4"); TEST_REQUIRE(info.at("details").at("parameter_size") == "4B"); - // Bookkeeping keys never reach the caller. + TEST_REQUIRE(info.at("backend") == "rai"); TEST_REQUIRE(!info.contains("supported_platforms")); - TEST_REQUIRE(!info.contains("platform_overrides")); - // And the entry names no backend: the hardware it was selected for is the - // backend, so there is nothing left for the catalog to say about it. - TEST_REQUIRE(!info.contains("supported_backends")); - TEST_REQUIRE(!info.at("details").contains("execution_backend")); + + // Having the kernels is not having the silicon. On aie2p the entry names + // another generation, so it is gone -- linking corelib does not make a + // corelib model appear on a machine that cannot run it -- while the + // FastFlowLM entries beside it are untouched. + auto corelib_on_aie2p = open_catalog("aie2p", {"flm", "rai"}); + TEST_REQUIRE(!corelib_on_aie2p.is_model_supported(kRaiTag)); + TEST_REQUIRE(corelib_on_aie2p.is_model_supported(kPhiTag)); + TEST_REQUIRE(corelib_on_aie2p.get_model_info(kPhiTag).second.at("backend") == "flm"); } -void test_pruned_lookups_do_not_throw() { - auto models = open_catalog("aie_next"); - // Both of these used to dereference the pruned llama3.2 family. - const auto [missing_tag, missing_info] = models.get_model_info("bogus:9b"); - TEST_REQUIRE(missing_tag == kPhiTag); - TEST_REQUIRE(missing_info.at("default_context_length") == 4096); - TEST_REQUIRE(models.rectify_model_tag("llama3.2") == "llama3.2"); - TEST_REQUIRE(models.rectify_model_tag("phi4-mini-it") == kPhiTag); +void test_a_catalog_can_filter_down_to_nothing() { + // The other corner of the same rule, and the one that reads like a bug if + // it is not stated: a build with only the FastFlowLM kernels, on silicon + // none of the FastFlowLM entries name. Every aie2p entry is pruned by the + // generation and the one aie_next entry by the kernels it would need, so + // the install offers nothing -- and that is a legible state, not a crash. + // It says so on stderr and keeps going; the commands fail one at a time, + // which is what makes `flm --help` still work on such a build. + auto nothing = open_catalog("aie_next", {"flm"}); + TEST_REQUIRE(nothing.all_tags.empty()); + TEST_REQUIRE(!nothing.is_model_supported(kPhiTag)); + TEST_REQUIRE(!nothing.is_model_supported(kRaiTag)); + TEST_REQUIRE(!nothing.is_model_supported("llama3.2:1b")); + // Asking anyway is an error with a reason, not a fallback to a model that + // is not there. + bool threw = false; + try { + (void)nothing.get_model_info(kPhiTag); + } catch (const std::exception&) { + threw = true; + } + TEST_REQUIRE(threw); +} + +void test_the_tag_name_decides_the_backend() { + // The rule, on a catalog small enough to see: a family ending in "-rai" is + // corelib's, anything else is the FastFlowLM kernels'. Nothing else in the + // entry is consulted, which is why the two never contend for a tag. + const auto path = write_catalog( + "flm_backend_model_list.json", + {{"model_path", "models"}, + {"models", + {{"demo", {{"1b", {{"name", "Demo"}}}}}, + {"demo-rai", {{"1b", {{"name", "Demo-rai"}}}}}}}}); + std::string list_path = path.string(); + std::string exe_dir = "."; + + // Neither entry says anything about silicon, so only the kernels decide: + // without the corelib ones the rai tag is not a model this binary can run, + // and the entries beside it are untouched. + model_list stock(list_path, exe_dir, "aie2p", {"flm"}); + TEST_REQUIRE(stock.is_model_supported("demo:1b")); + TEST_REQUIRE(!stock.is_model_supported("demo-rai:1b")); + TEST_REQUIRE(stock.get_model_info("demo:1b").second.at("backend") == "flm"); + // Both lookups used to dereference the pruned entry. A pruned family keeps + // its own name rather than resolving to a size, and an unknown tag comes + // back as something the caller can safely report on. + TEST_REQUIRE(stock.rectify_model_tag("demo-rai") == "demo-rai"); + TEST_REQUIRE(stock.rectify_model_tag("demo") == "demo:1b"); + const auto [missing_tag, missing_info] = stock.get_model_info("bogus:9b"); + TEST_REQUIRE(stock.is_model_supported(missing_tag)); + TEST_REQUIRE(missing_info.contains("name")); + + model_list corelib(list_path, exe_dir, "aie2p", {"flm", "rai"}); + TEST_REQUIRE(corelib.is_model_supported("demo:1b")); + TEST_REQUIRE(corelib.is_model_supported("demo-rai:1b")); + TEST_REQUIRE(corelib.get_model_info("demo-rai:1b").second.at("backend") == "rai"); + TEST_REQUIRE(corelib.get_model_info("demo:1b").second.at("backend") == "flm"); + + std::filesystem::remove(path); +} + +void test_an_entry_is_pruned_by_the_silicon_it_names() { + // The other axis, on its own: same build, same kernels, different machine. + const auto path = write_catalog( + "flm_platform_model_list.json", + {{"model_path", "models"}, + {"models", + {{"shipping", + {{"1b", {{"name", "Shipping"}, {"supported_platforms", {"aie2p"}}}}}}, + {"next", + {{"1b", {{"name", "Next"}, {"supported_platforms", {"aie_next"}}}}}}, + {"either", + {{"1b", + {{"name", "Either"}, + {"supported_platforms", {"aie2p", "aie_next"}}}}}}}}}); + std::string list_path = path.string(); + std::string exe_dir = "."; + + model_list aie2p(list_path, exe_dir, "aie2p", {"flm"}); + TEST_REQUIRE(aie2p.is_model_supported("shipping:1b")); + TEST_REQUIRE(aie2p.is_model_supported("either:1b")); + TEST_REQUIRE(!aie2p.is_model_supported("next:1b")); + // The key is bookkeeping: once the entry is kept, the generation is + // settled and nothing downstream has to know it was ever asked. + TEST_REQUIRE(!aie2p.get_model_info("shipping:1b").second.contains("supported_platforms")); + + model_list aie_next(list_path, exe_dir, "aie_next", {"flm"}); + TEST_REQUIRE(!aie_next.is_model_supported("shipping:1b")); + TEST_REQUIRE(aie_next.is_model_supported("either:1b")); + TEST_REQUIRE(aie_next.is_model_supported("next:1b")); + + std::filesystem::remove(path); +} + +void test_missing_key_means_every_generation() { + // Silence is the historical default, on both axes: an entry that names no + // silicon runs on all of it, and a tag that does not say "-rai" runs on the + // FastFlowLM kernels, which every family registers in every build. The + // shipped catalog still spells the platform out -- this is about catalogs + // older than the key, and about hand-made development trees. + const auto path = write_catalog( + "flm_legacy_model_list.json", + {{"model_path", "models"}, + {"models", {{"legacy", {{"1b", {{"name", "Legacy"}}}}}}}}); + std::string list_path = path.string(); + std::string exe_dir = "."; + + for (const char* platform : {"aie2p", "aie_next"}) { + model_list models(list_path, exe_dir, platform, {"flm"}); + TEST_REQUIRE(models.is_model_supported("legacy:1b")); + TEST_REQUIRE(models.get_model_info("legacy:1b").second.at("backend") == "flm"); + } + + std::filesystem::remove(path); } void test_platform_helpers() { - TEST_REQUIRE(utils::parse_platform("stx") == utils::npu_platform::stx); + TEST_REQUIRE(utils::parse_platform("aie2p") == utils::npu_platform::aie2p); TEST_REQUIRE(utils::parse_platform("aie_next") == utils::npu_platform::aie_next); TEST_REQUIRE(!utils::parse_platform("not-a-platform").has_value()); + TEST_REQUIRE(!utils::parse_platform("stx").has_value()); TEST_REQUIRE(utils::parse_platform(utils::platform_id(utils::npu_platform::aie_next)) == utils::npu_platform::aie_next); - TEST_REQUIRE(utils::default_npu_platform() == utils::npu_platform::stx); - - // The generation is whatever this binary was built for, nothing else. -#ifdef FLM_ENABLE_RAI - TEST_REQUIRE(utils::build_npu_platform() == utils::npu_platform::aie_next); -#else - TEST_REQUIRE(utils::build_npu_platform() == utils::npu_platform::stx); -#endif + TEST_REQUIRE(utils::default_npu_platform() == utils::npu_platform::aie_next); + + // The stand-in consults nothing -- not the environment, not which kernels + // were linked -- so it is the default here and in a corelib build alike. + // Changing what an install can see means changing default_npu_platform() + // and rebuilding, which is the one place to look. + TEST_REQUIRE(utils::get_device() == utils::default_npu_platform()); + TEST_REQUIRE(utils::get_device() == utils::get_device()); } void test_shipped_catalog_is_well_formed() { - const auto catalog = read_catalog(); - TEST_REQUIRE(!catalog.at("models").contains("phi4-mini-it-rai")); + const auto catalog = read_json(kCatalogPath); + bool saw_rai = false; for (const auto& [family, sizes] : catalog.at("models").items()) { + const bool rai_family = family.size() > 4 && + family.compare(family.size() - 4, 4, "-rai") == 0; for (const auto& [size, entry] : sizes.items()) { const std::string tag = family + ":" + size; - // Nothing in the catalog names a backend any more. - if (entry.contains("supported_backends")) { - throw std::runtime_error(tag + ": supported_backends is retired"); + // Retired spellings. Each of these once meant what + // "supported_platforms" plus the tag name now mean between them, + // and a catalog carrying one would be filtered on rules nothing + // reads. + for (const char* dead : {"platform_overrides", "supported_backend", + "supported_backends", "backend"}) { + if (entry.contains(dead)) { + throw std::runtime_error(tag + ": " + dead + " is retired"); + } } if (entry.contains("details") && entry.at("details").contains("execution_backend")) { throw std::runtime_error(tag + ": execution_backend is retired"); } - // An entry is only tagged if it runs somewhere other than stx, so - // the common case is no key at all. A key that says only ["stx"] - // is not wrong, just noise, and this keeps it from creeping back. - const nlohmann::json supported = - entry.value("supported_platforms", nlohmann::json::array()); - if (entry.contains("supported_platforms")) { - if (!supported.is_array() || supported.empty()) { - throw std::runtime_error(tag + ": supported_platforms must be a non-empty array"); - } - bool beyond_stx = false; - for (const auto& value : supported) { - if (!value.is_string() || - !utils::parse_platform(value.get()).has_value()) { - throw std::runtime_error(tag + ": unknown platform in supported_platforms"); - } - if (value.get() != "stx") beyond_stx = true; - } - if (!beyond_stx) { - throw std::runtime_error( - tag + ": supported_platforms says only stx, which is " - "the default -- drop the key"); - } + if (!entry.contains("supported_platforms")) { + throw std::runtime_error(tag + ": no supported_platforms"); } - if (!entry.contains("platform_overrides")) continue; - const auto& overrides = entry.at("platform_overrides"); - if (!overrides.is_object()) { - throw std::runtime_error(tag + ": platform_overrides must be an object"); + const auto& supported = entry.at("supported_platforms"); + if (!supported.is_array() || supported.empty()) { + throw std::runtime_error( + tag + ": supported_platforms must be a non-empty array"); } - for (const auto& [platform, patch] : overrides.items()) { - bool declared = false; - for (const auto& value : supported) { - if (value.get() == platform) declared = true; - } - if (!declared) { + std::set named; + for (const auto& value : supported) { + if (!value.is_string() || + !utils::parse_platform(value.get()).has_value()) { throw std::runtime_error( - tag + ": platform_overrides has '" + platform + - "', which is not in supported_platforms"); - } - if (!patch.is_object()) { - throw std::runtime_error(tag + ": override for '" + platform + - "' must be an object"); + tag + ": unknown platform in supported_platforms"); } + named.insert(value.get()); } - } - } -} + TEST_REQUIRE(named.size() <= kAllPlatforms.size()); -void test_missing_key_means_stx_only() { - // An untagged entry is stx-only. That is the rule the shipped catalog - // leans on -- only aie_next support gets a tag -- so it is worth pinning down - // from both sides: the untagged entry must appear on stx and must not - // leak onto aie_next. - const auto path = std::filesystem::temp_directory_path() / - "flm_legacy_model_list.json"; - nlohmann::json untagged = { - {"model_path", "models"}, - {"models", - {{"legacy", {{"1b", {{"name", "Legacy"}}}}}, - {"both", - {{"1b", - {{"name", "Both"}, - {"supported_platforms", {"stx", "aie_next"}}}}}}}}}; - { - std::ofstream out(path); - out << untagged.dump(2); + // corelib runs on the next generation and the FastFlowLM kernels on + // what is shipping, so a tag that says one and a platform key that + // says the other describes a package nothing can run. + if (rai_family != (named.count("aie_next") != 0)) { + throw std::runtime_error( + tag + ": the tag name and supported_platforms disagree " + "about which generation this package is for"); + } + if (rai_family) saw_rai = true; + } } - std::string list_path = path.string(); - std::string exe_dir = "."; - model_list stx(list_path, exe_dir, "stx"); - TEST_REQUIRE(stx.is_model_supported("legacy:1b")); - TEST_REQUIRE(stx.is_model_supported("both:1b")); - - model_list aie_next(list_path, exe_dir, "aie_next"); - TEST_REQUIRE(!aie_next.is_model_supported("legacy:1b")); - TEST_REQUIRE(aie_next.is_model_supported("both:1b")); - - std::filesystem::remove(path); + // A corelib model is packaged differently from its FastFlowLM namesake, so + // it has its own tag; if that stops being true, everything above is + // checking a rule the catalog no longer follows. + TEST_REQUIRE(saw_rai); + TEST_REQUIRE(catalog.at("models").contains("phi4-mini-it")); + TEST_REQUIRE(catalog.at("models").contains("phi4-mini-it-rai")); } } // namespace int main() { - RunTest(test_stx_entry_is_unchanged, "stx entry is unchanged"); - RunTest(test_aie_next_entry_is_merged, "aie_next entry is merged"); - RunTest(test_pruned_lookups_do_not_throw, "pruned lookups do not throw"); + RunTest(test_stock_build_offers_phi4_with_its_own_artifacts, + "stock build offers phi4 with its own artifacts"); + RunTest(test_a_corelib_build_offers_the_corelib_phi4, + "a corelib build offers the corelib phi4"); + RunTest(test_the_tag_name_decides_the_backend, + "the tag name decides the backend"); + RunTest(test_a_catalog_can_filter_down_to_nothing, + "a catalog can filter down to nothing"); + RunTest(test_an_entry_is_pruned_by_the_silicon_it_names, + "an entry is pruned by the silicon it names"); + RunTest(test_missing_key_means_every_generation, + "missing key means every generation"); RunTest(test_platform_helpers, "platform helpers"); - RunTest(test_shipped_catalog_is_well_formed, "shipped catalog is well formed"); - RunTest(test_missing_key_means_stx_only, "missing key means stx only"); + RunTest(test_shipped_catalog_is_well_formed, + "shipped catalog is well formed"); std::cout << "All model_list platform tests passed\n"; return 0; } diff --git a/src/test/phi4_rai/test_model_downloader.cpp b/src/test/phi4_rai/test_model_downloader.cpp index 1d4c1e15f..bd2006a62 100644 --- a/src/test/phi4_rai/test_model_downloader.cpp +++ b/src/test/phi4_rai/test_model_downloader.cpp @@ -12,24 +12,29 @@ namespace { namespace fs = std::filesystem; -// One tag now covers both NPU generations; the catalog picks the artifacts. -constexpr const char* kRaiTag = "phi4-mini-it:4b"; -// The rai build still installs under its own directory and reads its own -// model_info.json record set, so those two names stay distinct from the tag. +// The corelib model is packaged differently from its FastFlowLM namesake -- +// a Q8_0 GGUF against an NPU2/Q4NX build -- so it is a separate tag, and the +// "-rai" suffix is what routes it to the corelib kernels. +constexpr const char* kRaiTag = "phi4-mini-it-rai:4b"; +constexpr const char* kFlmTag = "phi4-mini-it:4b"; +// The tag is the model_info.json key and the directory name, so all three read +// alike and nothing has to redirect between them. constexpr const char* kRaiModelInfoKey = "phi4-mini-it-rai:4b"; constexpr const char* kRaiDirName = "phi4-mini-it-rai"; constexpr const char* kUnslothRevision = "78eb92a46fc37e6b524df991ed9aca9bc6aa7b80"; constexpr const char* kMicrosoftRevision = "cfbefacb99257ffa30c83adab238a50856ac3083"; -/// \brief the shipped catalog entry as one platform resolves it -/// \param platform "stx" or "aie_next" +/// \brief the shipped catalog entry as the machine that can run it resolves it /// \param tag the model tag to resolve -/// \note Goes through model_list so these tests exercise the real -/// filter-and-merge path rather than the raw JSON. -nlohmann::json ResolvedModel(const std::string& platform, const char* tag) { +/// \param platform the NPU generation to resolve it for +/// \note Goes through model_list so these tests exercise the real filtering +/// path rather than the raw JSON, with a corelib build's backend list -- +/// a tag whose flow this build lacks, or whose silicon this is not, is +/// pruned. +nlohmann::json ResolvedModel(const char* tag, const char* platform) { std::string path = FLM_SOURCE_DIR "/model_list.json"; std::string exe_dir = "."; - model_list models(path, exe_dir, platform); + model_list models(path, exe_dir, platform, {"flm", "rai"}); TEST_REQUIRE(models.is_model_supported(tag)); return models.get_model_info(tag).second; } @@ -79,25 +84,28 @@ std::string FileUrl(const fs::path& path) { } void TestRaiCatalogHasExactlyFourFilesAndExpectedDirectoryName() { - const auto model = ResolvedModel("aie_next", kRaiTag); + const auto model = ResolvedModel(kRaiTag, "aie_next"); const std::vector expected = { "Phi-4-mini-instruct.Q8_0.gguf", "tokenizer.json", "tokenizer_config.json", "config.json"}; TEST_REQUIRE(model.at("name") == kRaiDirName); - TEST_REQUIRE(model.at("model_info_key") == kRaiModelInfoKey); + // The tag is the key, so there is nothing to redirect. + TEST_REQUIRE(!model.contains("model_info_key")); + TEST_REQUIRE(std::string(kRaiTag) == kRaiModelInfoKey); + TEST_REQUIRE(model.at("backend") == "rai"); TEST_REQUIRE(model.at("files").get>() == expected); TEST_REQUIRE(model.at("size").get() == 4100140571ULL); } void TestGgufUrlContainsUnslothRevisionAndFilename() { - const auto model = ResolvedModel("aie_next", kRaiTag); + const auto model = ResolvedModel(kRaiTag, "aie_next"); const auto source = resolve_file_source(model, "Phi-4-mini-instruct.Q8_0.gguf", false); TEST_REQUIRE(source.url == std::string("https://huggingface.co/unsloth/Phi-4-mini-instruct-GGUF/resolve/") + kUnslothRevision + "/Phi-4-mini-instruct.Q8_0.gguf?download=true"); } void TestThreeFrontendUrlsContainMicrosoftRevisionAndFilename() { - const auto model = ResolvedModel("aie_next", kRaiTag); + const auto model = ResolvedModel(kRaiTag, "aie_next"); for (const std::string filename : {"tokenizer.json", "tokenizer_config.json", "config.json"}) { const auto source = resolve_file_source(model, filename, false); TEST_REQUIRE(source.url == std::string("https://huggingface.co/microsoft/Phi-4-mini-instruct/resolve/") + @@ -106,8 +114,10 @@ void TestThreeFrontendUrlsContainMicrosoftRevisionAndFilename() { } void TestExistingSingleSourceEntryKeepsItsCurrentUrl() { - // The same tag on stx: the aie_next override must not leak onto Strix. - const auto model = ResolvedModel("stx", kRaiTag); + // The FastFlowLM phi4, which a corelib build still lists and still runs on + // the flm kernels: nothing about the corelib tag may reach it. + const auto model = ResolvedModel(kFlmTag, "aie2p"); + TEST_REQUIRE(model.at("backend") == "flm"); TEST_REQUIRE(model.at("name") == "Phi4-mini-Instruct-NPU2"); TEST_REQUIRE(!model.contains("file_sources")); TEST_REQUIRE(!model.contains("model_info_key")); @@ -136,17 +146,19 @@ void TestUnknownFileSourceKeyAndMissingUrlOrRevisionFail() { void TestActualRaiCatalogTreatsPinnedConfigWithoutFlmVersionAsCompatible() { const auto root = TempDirectory("actual-catalog-version"); - // Take the merged aie_next entry and re-home it in a temp catalog. It carries no - // supported_platforms any more, so the default (stx) constructor keeps it. - const auto model = ResolvedModel("aie_next", kRaiTag); + // Take the resolved entry and re-home it in a temp catalog, under the same + // family: the tag is what says these are corelib artifacts. + const auto model = ResolvedModel(kRaiTag, "aie_next"); const nlohmann::json catalog = { {"model_path", "models"}, - {"models", {{"phi4-mini-it", {{"4b", model}}}}}}; + {"models", {{"phi4-mini-it-rai", {{"4b", model}}}}}}; const auto catalog_path = root / "model_list.json"; Write(catalog_path, catalog.dump()); std::string catalog_string = catalog_path.string(); std::string root_string = root.string(); - model_list models(catalog_string, root_string); + // The tag names the corelib flow and the entry the next generation, so say + // both are here. + model_list models(catalog_string, root_string, "aie_next", {"flm", "rai"}); const auto model_path = root / "models" / kRaiDirName; for (const auto& filename : model.at("files")) { Write(model_path / filename.get(), "placeholder"); From bebaf793a9092bf716e45af822787d911cda0124 Mon Sep 17 00:00:00 2001 From: alfxu_amdeng Date: Thu, 24 Sep 2026 10:50:29 -0700 Subject: [PATCH 07/17] refactor(models): put backend and platform in every model path Model code sat at models//, with rai sources at models//rai/. That encodes at most one of the two axes a model actually varies over, and encodes neither for the flm engines: nothing in the path said which silicon a header was written against. With aie_next and rai both arriving, a second provider or a second generation for a family had nowhere to go but a conditional. Both axes now appear in the path, family first: models//// So models/phi4/flm/aie2p/phi4_npu.hpp beside models/phi4/rai/aie_next/ phi4_rai.hpp -- one family directory holding both providers, each naming the generation it was written for. Neither axis is derived from the other; the path just records both answers. Pure relocation: 43 files moved as renames, every #include and build reference repointed, no code changed. models_sources.cmake globs */rai/*/*.cpp to reach through the new platform level, and the layout sections of common/models/README.md and create_new_model.md describe the three levels. Verified: cmake configure + ninja build clean with -DFLM_ENABLE_RAI=ON (64/64, flm links and runs), the glob picks up all six rai sources, and every models/ include resolves to a file on disk. Co-Authored-By: Claude Opus 5 --- .../modeling_gemma_embedding.cpp | 2 +- src/common/AutoModel/builtin_backends.cpp | 2 +- src/common/AutoModel/modeling_hunyuan.cpp | 2 +- src/common/models/README.md | 35 ++++++++++-------- src/common/models/models_sources.cmake | 9 ++--- .../phi4/rai/{ => aie_next}/phi4_rai.cpp | 10 +++--- .../rai/{ => aie_next}/phi4_rai_backend.cpp | 6 ++-- .../phi4/rai/{ => aie_next}/phi4_rai_gguf.cpp | 4 +-- .../phi4/rai/{ => aie_next}/phi4_rai_host.cpp | 4 +-- .../{ => aie_next}/phi4_rai_shape_plan.cpp | 4 +-- .../{ => aie_next}/phi4_rai_weight_cache.cpp | 2 +- src/create_new_model.md | 14 +++++--- .../modeling_gemma_embedding.hpp | 2 +- src/include/AutoModel/automodel.hpp | 36 +++++++++---------- .../AutoModel/modeling_qwen3_5_omni.hpp | 2 +- .../gemma/{ => flm/aie2p}/gemma_npu.hpp | 0 .../{ => flm/aie2p}/gemma_npu_sequence.hpp | 0 .../{ => flm/aie2p}/gemma4_12b_npu.hpp | 0 .../gemma4e/{ => flm/aie2p}/gemma4e_npu.hpp | 0 .../{ => flm/aie2p}/gemma4e_flash.hpp | 2 +- .../{ => flm/aie2p}/gemma_embedding.hpp | 0 .../{ => flm/aie2p}/gemma_text_dequant.hpp | 0 .../{ => flm/aie2p}/gemma_text_gemm.hpp | 0 .../{ => flm/aie2p}/gemma_text_lm_head.hpp | 0 .../{ => flm/aie2p}/gemma_text_npu.hpp | 0 .../aie2p}/gemma_text_npu_sequence.hpp | 0 .../gpt_oss/{ => flm/aie2p}/gpt_oss_npu.hpp | 0 .../{ => flm/aie2p}/gpt_oss_npu_sequence.hpp | 0 .../hunyuan/{ => flm/aie2p}/hunyuan_npu.hpp | 0 .../models/lfm2/{ => flm/aie2p}/lfm2_npu.hpp | 0 .../llama/{ => flm/aie2p}/llama_npu.hpp | 0 .../{ => flm/aie2p}/llama_npu_sequence.hpp | 0 .../nanbeige/{ => flm/aie2p}/nanbeige_npu.hpp | 2 +- .../{ => flm/aie2p}/nanbeige_npu_sequence.hpp | 0 .../models/phi4/{ => flm/aie2p}/phi4_npu.hpp | 0 .../{ => flm/aie2p}/phi4_npu_sequence.hpp | 0 .../phi4/rai/{ => aie_next}/phi4_rai.hpp | 2 +- .../rai/{ => aie_next}/phi4_rai_backend.hpp | 0 .../rai/{ => aie_next}/phi4_rai_constants.hpp | 0 .../phi4/rai/{ => aie_next}/phi4_rai_gguf.hpp | 0 .../phi4/rai/{ => aie_next}/phi4_rai_host.hpp | 2 +- .../{ => aie_next}/phi4_rai_shape_plan.hpp | 0 .../{ => aie_next}/phi4_rai_weight_cache.hpp | 0 .../qwen2/{ => flm/aie2p}/qwen2_npu.hpp | 0 .../qwen2vl/{ => flm/aie2p}/qwen2vl_npu.hpp | 0 .../qwen3/{ => flm/aie2p}/qwen3_npu.hpp | 0 .../{ => flm/aie2p}/qwen3_npu_sequence.hpp | 0 .../{ => flm/aie2p}/qwen3_5_omni.hpp | 0 .../{ => flm/aie2p}/qwen3_5vl_npu.hpp | 0 .../{ => flm/aie2p}/qwen3_6_moe_npu.hpp | 0 .../qwen3vl/{ => flm/aie2p}/qwen3vl_npu.hpp | 0 .../{ => flm/aie2p}/qwen3vl_flash.hpp | 2 +- src/src/main.cpp | 16 +++++---- src/test/gemma4_12b/Makefile | 2 +- src/test/gemma4e_flash/Makefile | 4 +-- src/test/hunyuan_npu/Makefile | 2 +- src/test/phi4_rai/CMakeLists.txt | 20 +++++------ src/test/phi4_rai/test_phi4_engine.cpp | 8 ++--- src/test/phi4_rai/test_phi4_frontend.cpp | 4 +-- src/test/phi4_rai/test_phi4_gguf.cpp | 4 +-- src/test/phi4_rai/test_phi4_host.cpp | 2 +- src/test/phi4_rai/test_phi4_shape_plan.cpp | 4 +-- src/test/qwen3_5_omni_npu/Makefile | 2 +- src/test/qwen3vl_flash/Makefile | 4 +-- 64 files changed, 115 insertions(+), 101 deletions(-) rename src/common/models/phi4/rai/{ => aie_next}/phi4_rai.cpp (99%) rename src/common/models/phi4/rai/{ => aie_next}/phi4_rai_backend.cpp (96%) rename src/common/models/phi4/rai/{ => aie_next}/phi4_rai_gguf.cpp (99%) rename src/common/models/phi4/rai/{ => aie_next}/phi4_rai_host.cpp (98%) rename src/common/models/phi4/rai/{ => aie_next}/phi4_rai_shape_plan.cpp (97%) rename src/common/models/phi4/rai/{ => aie_next}/phi4_rai_weight_cache.cpp (99%) rename src/include/models/gemma/{ => flm/aie2p}/gemma_npu.hpp (100%) rename src/include/models/gemma/{ => flm/aie2p}/gemma_npu_sequence.hpp (100%) rename src/include/models/gemma4_12b/{ => flm/aie2p}/gemma4_12b_npu.hpp (100%) rename src/include/models/gemma4e/{ => flm/aie2p}/gemma4e_npu.hpp (100%) rename src/include/models/gemma4e_flash/{ => flm/aie2p}/gemma4e_flash.hpp (99%) rename src/include/models/gemma_embedding/{ => flm/aie2p}/gemma_embedding.hpp (100%) rename src/include/models/gemma_text/{ => flm/aie2p}/gemma_text_dequant.hpp (100%) rename src/include/models/gemma_text/{ => flm/aie2p}/gemma_text_gemm.hpp (100%) rename src/include/models/gemma_text/{ => flm/aie2p}/gemma_text_lm_head.hpp (100%) rename src/include/models/gemma_text/{ => flm/aie2p}/gemma_text_npu.hpp (100%) rename src/include/models/gemma_text/{ => flm/aie2p}/gemma_text_npu_sequence.hpp (100%) rename src/include/models/gpt_oss/{ => flm/aie2p}/gpt_oss_npu.hpp (100%) rename src/include/models/gpt_oss/{ => flm/aie2p}/gpt_oss_npu_sequence.hpp (100%) rename src/include/models/hunyuan/{ => flm/aie2p}/hunyuan_npu.hpp (100%) rename src/include/models/lfm2/{ => flm/aie2p}/lfm2_npu.hpp (100%) rename src/include/models/llama/{ => flm/aie2p}/llama_npu.hpp (100%) rename src/include/models/llama/{ => flm/aie2p}/llama_npu_sequence.hpp (100%) rename src/include/models/nanbeige/{ => flm/aie2p}/nanbeige_npu.hpp (97%) rename src/include/models/nanbeige/{ => flm/aie2p}/nanbeige_npu_sequence.hpp (100%) rename src/include/models/phi4/{ => flm/aie2p}/phi4_npu.hpp (100%) rename src/include/models/phi4/{ => flm/aie2p}/phi4_npu_sequence.hpp (100%) rename src/include/models/phi4/rai/{ => aie_next}/phi4_rai.hpp (96%) rename src/include/models/phi4/rai/{ => aie_next}/phi4_rai_backend.hpp (100%) rename src/include/models/phi4/rai/{ => aie_next}/phi4_rai_constants.hpp (100%) rename src/include/models/phi4/rai/{ => aie_next}/phi4_rai_gguf.hpp (100%) rename src/include/models/phi4/rai/{ => aie_next}/phi4_rai_host.hpp (93%) rename src/include/models/phi4/rai/{ => aie_next}/phi4_rai_shape_plan.hpp (100%) rename src/include/models/phi4/rai/{ => aie_next}/phi4_rai_weight_cache.hpp (100%) rename src/include/models/qwen2/{ => flm/aie2p}/qwen2_npu.hpp (100%) rename src/include/models/qwen2vl/{ => flm/aie2p}/qwen2vl_npu.hpp (100%) rename src/include/models/qwen3/{ => flm/aie2p}/qwen3_npu.hpp (100%) rename src/include/models/qwen3/{ => flm/aie2p}/qwen3_npu_sequence.hpp (100%) rename src/include/models/qwen3_5_omni/{ => flm/aie2p}/qwen3_5_omni.hpp (100%) rename src/include/models/qwen3_5vl/{ => flm/aie2p}/qwen3_5vl_npu.hpp (100%) rename src/include/models/qwen3_6_moe/{ => flm/aie2p}/qwen3_6_moe_npu.hpp (100%) rename src/include/models/qwen3vl/{ => flm/aie2p}/qwen3vl_npu.hpp (100%) rename src/include/models/qwen3vl_flash/{ => flm/aie2p}/qwen3vl_flash.hpp (97%) diff --git a/src/common/AutoEmbeddingModel/modeling_gemma_embedding.cpp b/src/common/AutoEmbeddingModel/modeling_gemma_embedding.cpp index 375c0b0b2..90f6df0e2 100644 --- a/src/common/AutoEmbeddingModel/modeling_gemma_embedding.cpp +++ b/src/common/AutoEmbeddingModel/modeling_gemma_embedding.cpp @@ -6,7 +6,7 @@ /// \note This is a source file for the Gemma_Embedding class #include "AutoEmbeddingModel/modeling_gemma_embedding.hpp" -#include "models/gemma_embedding/gemma_embedding.hpp" +#include "models/gemma_embedding/flm/aie2p/gemma_embedding.hpp" Gemma_Embedding::Gemma_Embedding(flm_rt::device* npu_device_inst) : AutoEmbeddingModel(npu_device_inst, "embed-gemma:300m") { } diff --git a/src/common/AutoModel/builtin_backends.cpp b/src/common/AutoModel/builtin_backends.cpp index 0fef482d9..0e18742ab 100644 --- a/src/common/AutoModel/builtin_backends.cpp +++ b/src/common/AutoModel/builtin_backends.cpp @@ -9,7 +9,7 @@ #include "AutoModel/model_backend.hpp" #if defined(FLM_ENABLE_RAI) -#include "models/phi4/rai/phi4_rai_backend.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_backend.hpp" #endif namespace flm::backend { diff --git a/src/common/AutoModel/modeling_hunyuan.cpp b/src/common/AutoModel/modeling_hunyuan.cpp index 2289b7060..32de5ad8e 100644 --- a/src/common/AutoModel/modeling_hunyuan.cpp +++ b/src/common/AutoModel/modeling_hunyuan.cpp @@ -6,7 +6,7 @@ /// \note AutoModel wrapper for the `hunyuan-dense` engine (Hy-MT2-1.8B). #include "AutoModel/modeling_hunyuan.hpp" -#include "models/hunyuan/hunyuan_npu.hpp" +#include "models/hunyuan/flm/aie2p/hunyuan_npu.hpp" /************ hunyuan-dense family **************/ Hunyuan::Hunyuan(flm_rt::device* npu_device_inst) : AutoModel(npu_device_inst, "Hunyuan") { diff --git a/src/common/models/README.md b/src/common/models/README.md index 16a47b971..92f502be1 100644 --- a/src/common/models/README.md +++ b/src/common/models/README.md @@ -81,20 +81,27 @@ entries, not in the backend id. ## 2. Lay out the files ``` -src/include/models//rai/ headers for the corelib implementation -src/common/models//rai/ the corelib implementation +src/include/models//// headers +src/common/models//// the implementation ``` -The folders are named after the kernel provider, not the silicon. FastFlowLM's -own engines have no folder here — they ship as prebuilt libraries under -`lib//` and their headers stay at `models//`. Headers mirror the -source split, so an `#include` says which provider it belongs to -(`models/phi4/rai/phi4_rai_gguf.hpp` against `models/phi4/phi4_npu.hpp`), and a -grep for `models//rai/` finds everything the rai path pulls in. +Both axes from the top of this document appear in the path, in that order: the +family owns the directory, the kernel provider comes next, and the silicon +generation last — `phi4/rai/aie_next/` for the corelib implementation, and +`phi4/flm/aie2p/` for the FastFlowLM one. Nothing keys one axis off the other; +the path just records both answers, so a family that gains a second provider or +a second generation grows a sibling folder instead of a conditional. + +FastFlowLM's own engines have no *sources* here — they ship as prebuilt libraries +under `lib//` — but their headers follow the same three levels. Headers +mirror the source split, so an `#include` says which provider and which silicon +it belongs to (`models/phi4/rai/aie_next/phi4_rai_gguf.hpp` against +`models/phi4/flm/aie2p/phi4_npu.hpp`), and a grep for `models//rai/` finds +everything the rai path pulls in. **Do not edit any `CMakeLists.txt` for this.** [`models_sources.cmake`](models_sources.cmake) -globs `*/rai/*.cpp`, and that glob is what `flm_rai` compiles. Creating the -folder is the whole registration step. +globs `*/rai/*/*.cpp` — every platform under every family's rai folder — and that +glob is what `flm_rai` compiles. Creating the folder is the whole registration step. Phi-4's rai side is five translation units, and the split is worth copying: @@ -113,7 +120,7 @@ without hardware. ## 3. The engine: a `causal_lm` subclass -Model it on [`phi4_rai.hpp`](../../include/models/phi4/rai/phi4_rai.hpp). +Model it on [`phi4_rai.hpp`](../../include/models/phi4/rai/aie_next/phi4_rai.hpp). Two rules matter more than the rest. ### `causal_lm.hpp` is a frozen ABI @@ -158,7 +165,7 @@ engine type, never through a `causal_lm*`. (`kWeightCreateConcurrency`). The per-create thread hint (`kRequantizeThreads`) stays at corelib's default of one so the two forms of parallelism do not multiply into an oversubscribed machine. See - [`phi4_rai_constants.hpp`](../../include/models/phi4/rai/phi4_rai_constants.hpp). + [`phi4_rai_constants.hpp`](../../include/models/phi4/rai/aie_next/phi4_rai_constants.hpp). - Expose `bool poisoned() const noexcept`. A corelib failure mid-decode usually leaves device state that only a reload can clear; the backend surfaces this and `AutoModel` turns it into a 500 that asks for an unload/reload. @@ -172,7 +179,7 @@ engine type, never through a `causal_lm*`. [`ModelBackend`](../../include/AutoModel/model_backend.hpp) owns one engine **and every rule for driving it**. Its defaults describe the FastFlowLM NPU engines, so you override only what differs. Phi-4's corelib backend -([`phi4_rai_backend.cpp`](phi4/rai/phi4_rai_backend.cpp)) overrides six: +([`phi4_rai_backend.cpp`](phi4/rai/aie_next/phi4_rai_backend.cpp)) overrides six: | override | corelib value | why | |---|---|---| @@ -373,7 +380,7 @@ Check, in order: ## Checklist -- [ ] `src/common/models//rai/` created (no CMake edit) +- [ ] `src/common/models//rai//` created (no CMake edit) - [ ] GGUF/host layers hold no corelib types - [ ] `load_weights` is a documented throwing shim - [ ] weight creates are serialized, with the thread hint passed diff --git a/src/common/models/models_sources.cmake b/src/common/models/models_sources.cmake index 41370118d..c2808c16b 100644 --- a/src/common/models/models_sources.cmake +++ b/src/common/models/models_sources.cmake @@ -1,7 +1,8 @@ -# Per-model sources, split by where the kernels come from. A model that reaches -# its kernels through ryzenai-corelib puts them in: -# /rai/ - built into flm_rai when FLM_ENABLE_RAI is on +# Per-model sources, split by where the kernels come from and which silicon they +# run on. Those two axes are independent, so both appear in the path: +# /// - e.g. phi4/rai/aie_next/ +# The rai sources are built into flm_rai when FLM_ENABLE_RAI is on. # FastFlowLM's own flow has no per-model sources here: those engines ship as # prebuilt libraries under lib//. # Adding a model means adding the folder, not editing this file. -file(GLOB FLM_MODELS_RAI_SOURCES "${CMAKE_CURRENT_LIST_DIR}/*/rai/*.cpp") +file(GLOB FLM_MODELS_RAI_SOURCES "${CMAKE_CURRENT_LIST_DIR}/*/rai/*/*.cpp") diff --git a/src/common/models/phi4/rai/phi4_rai.cpp b/src/common/models/phi4/rai/aie_next/phi4_rai.cpp similarity index 99% rename from src/common/models/phi4/rai/phi4_rai.cpp rename to src/common/models/phi4/rai/aie_next/phi4_rai.cpp index 18652dedf..87f1e5e18 100644 --- a/src/common/models/phi4/rai/phi4_rai.cpp +++ b/src/common/models/phi4/rai/aie_next/phi4_rai.cpp @@ -1,9 +1,9 @@ -#include "models/phi4/rai/phi4_rai.hpp" +#include "models/phi4/rai/aie_next/phi4_rai.hpp" #include "rai/corelib_object.hpp" -#include "models/phi4/rai/phi4_rai_constants.hpp" -#include "models/phi4/rai/phi4_rai_host.hpp" -#include "models/phi4/rai/phi4_rai_shape_plan.hpp" -#include "models/phi4/rai/phi4_rai_weight_cache.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_constants.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_host.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_shape_plan.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_weight_cache.hpp" #include #include #include diff --git a/src/common/models/phi4/rai/phi4_rai_backend.cpp b/src/common/models/phi4/rai/aie_next/phi4_rai_backend.cpp similarity index 96% rename from src/common/models/phi4/rai/phi4_rai_backend.cpp rename to src/common/models/phi4/rai/aie_next/phi4_rai_backend.cpp index 3cd58f7ac..86e08c25b 100644 --- a/src/common/models/phi4/rai/phi4_rai_backend.cpp +++ b/src/common/models/phi4/rai/aie_next/phi4_rai_backend.cpp @@ -1,10 +1,10 @@ /// \file phi4_rai_backend.cpp /// \brief The ryzenai-corelib backend for Phi-4 -#include "models/phi4/rai/phi4_rai_backend.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_backend.hpp" #include "rai/corelib_runtime.hpp" -#include "models/phi4/rai/phi4_rai.hpp" -#include "models/phi4/rai/phi4_rai_gguf.hpp" +#include "models/phi4/rai/aie_next/phi4_rai.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_gguf.hpp" #include "utils/file_access.hpp" #include "utils/utils.hpp" diff --git a/src/common/models/phi4/rai/phi4_rai_gguf.cpp b/src/common/models/phi4/rai/aie_next/phi4_rai_gguf.cpp similarity index 99% rename from src/common/models/phi4/rai/phi4_rai_gguf.cpp rename to src/common/models/phi4/rai/aie_next/phi4_rai_gguf.cpp index bf8061a93..c6b81c933 100644 --- a/src/common/models/phi4/rai/phi4_rai_gguf.cpp +++ b/src/common/models/phi4/rai/aie_next/phi4_rai_gguf.cpp @@ -1,6 +1,6 @@ -#include "models/phi4/rai/phi4_rai_gguf.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_gguf.hpp" -#include "models/phi4/rai/phi4_rai_constants.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_constants.hpp" #include "utils/file_access.hpp" #ifdef _WIN32 diff --git a/src/common/models/phi4/rai/phi4_rai_host.cpp b/src/common/models/phi4/rai/aie_next/phi4_rai_host.cpp similarity index 98% rename from src/common/models/phi4/rai/phi4_rai_host.cpp rename to src/common/models/phi4/rai/aie_next/phi4_rai_host.cpp index 1c3abbb89..2f431f4e1 100644 --- a/src/common/models/phi4/rai/phi4_rai_host.cpp +++ b/src/common/models/phi4/rai/aie_next/phi4_rai_host.cpp @@ -1,6 +1,6 @@ -#include "models/phi4/rai/phi4_rai_host.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_host.hpp" -#include "models/phi4/rai/phi4_rai_constants.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_constants.hpp" #include #include diff --git a/src/common/models/phi4/rai/phi4_rai_shape_plan.cpp b/src/common/models/phi4/rai/aie_next/phi4_rai_shape_plan.cpp similarity index 97% rename from src/common/models/phi4/rai/phi4_rai_shape_plan.cpp rename to src/common/models/phi4/rai/aie_next/phi4_rai_shape_plan.cpp index 529322e98..a5b5c76ea 100644 --- a/src/common/models/phi4/rai/phi4_rai_shape_plan.cpp +++ b/src/common/models/phi4/rai/aie_next/phi4_rai_shape_plan.cpp @@ -1,6 +1,6 @@ -#include "models/phi4/rai/phi4_rai_shape_plan.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_shape_plan.hpp" -#include "models/phi4/rai/phi4_rai_constants.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_constants.hpp" #include #include diff --git a/src/common/models/phi4/rai/phi4_rai_weight_cache.cpp b/src/common/models/phi4/rai/aie_next/phi4_rai_weight_cache.cpp similarity index 99% rename from src/common/models/phi4/rai/phi4_rai_weight_cache.cpp rename to src/common/models/phi4/rai/aie_next/phi4_rai_weight_cache.cpp index 4b9f942ae..b82f0bcb4 100644 --- a/src/common/models/phi4/rai/phi4_rai_weight_cache.cpp +++ b/src/common/models/phi4/rai/aie_next/phi4_rai_weight_cache.cpp @@ -1,6 +1,6 @@ /// \file phi4_rai_weight_cache.cpp /// \brief On-disk cache of the packed weights -#include "models/phi4/rai/phi4_rai_weight_cache.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_weight_cache.hpp" #include diff --git a/src/create_new_model.md b/src/create_new_model.md index 681a606d8..73c1dbea8 100644 --- a/src/create_new_model.md +++ b/src/create_new_model.md @@ -35,9 +35,13 @@ Every model has two layers: Before writing any wrapper code, these artifacts must exist: -- **Engine header** — `include/models//_npu.hpp` +- **Engine header** — `include/models//flm/aie2p/_npu.hpp` Declares `class _npu : public causal_lm` plus any model-specific payload - structs (image / audio descriptors for omni models). + structs (image / audio descriptors for omni models). The path carries both axes + after the family: `flm` is the kernel provider (FastFlowLM's own flow) and + `aie2p` the silicon generation. A model reaching its kernels through corelib on + the newer generation sits at `/rai/aie_next/` instead; see + [`common/models/README.md`](common/models/README.md). - **Engine library** - Linux: `lib/xrt/lib_npu.so` and, if the HRX runtime is used, `lib/hrx/lib_npu.so` @@ -195,8 +199,8 @@ by `<|"|>` that need a normalizing rewrite pass before `nlohmann::json::parse`. `include/AutoModel/automodel.hpp` — add the engine include next to the others: ```cpp -#include "models/gemma4e/gemma4e_npu.hpp" -#include "models/gemma4_12b/gemma4_12b_npu.hpp" +#include "models/gemma4e/flm/aie2p/gemma4e_npu.hpp" +#include "models/gemma4_12b/flm/aie2p/gemma4_12b_npu.hpp" ``` > Watch for symbol collisions with sibling engines (e.g. `is_swa_layer` overloads). @@ -429,7 +433,7 @@ flm serve gemma4-12b:12b # then exercise /v1/chat/completions, streaming, an ## Checklist -- [ ] `include/models//_npu.hpp` present, exports verified with `nm -DC` +- [ ] `include/models//flm/aie2p/_npu.hpp` present, exports verified with `nm -DC` - [ ] `lib/xrt/lib_npu.so` (and `lib/hrx/`, `.dll` + `.lib` for Windows) - [ ] `xclbins//` present, name matches `model_list.json` - [ ] `include/AutoModel/modeling_.hpp` diff --git a/src/include/AutoEmbeddingModel/modeling_gemma_embedding.hpp b/src/include/AutoEmbeddingModel/modeling_gemma_embedding.hpp index d0ef9d72e..dcc986167 100644 --- a/src/include/AutoEmbeddingModel/modeling_gemma_embedding.hpp +++ b/src/include/AutoEmbeddingModel/modeling_gemma_embedding.hpp @@ -6,7 +6,7 @@ /// \note This is a header file for the Gemma_Embedding class #include "auto_embedding_model.hpp" -#include "models/gemma_embedding/gemma_embedding.hpp" +#include "models/gemma_embedding/flm/aie2p/gemma_embedding.hpp" class Gemma_Embedding : public AutoEmbeddingModel{ private: diff --git a/src/include/AutoModel/automodel.hpp b/src/include/AutoModel/automodel.hpp index 5708b17f3..b301a819c 100644 --- a/src/include/AutoModel/automodel.hpp +++ b/src/include/AutoModel/automodel.hpp @@ -23,24 +23,24 @@ #include "causal_lm.hpp" #include "lm_config.hpp" #include "AutoModel/model_backend.hpp" -#include "models/llama/llama_npu.hpp" -#include "models/qwen2/qwen2_npu.hpp" -#include "models/qwen3/qwen3_npu.hpp" -#include "models/qwen2vl/qwen2vl_npu.hpp" -#include "models/qwen3vl/qwen3vl_npu.hpp" -#include "models/qwen3vl_flash/qwen3vl_flash.hpp" -#include "models/qwen3_5vl/qwen3_5vl_npu.hpp" -#include "models/qwen3_6_moe/qwen3_6_moe_npu.hpp" -#include "models/gemma/gemma_npu.hpp" -#include "models/gemma_text/gemma_text_npu.hpp" -#include "models/gemma4e/gemma4e_npu.hpp" -#include "models/gemma4e_flash/gemma4e_flash.hpp" -#include "models/gemma4_12b/gemma4_12b_npu.hpp" -#include "models/lfm2/lfm2_npu.hpp" -#include "models/phi4/phi4_npu.hpp" -#include "models/gpt_oss/gpt_oss_npu.hpp" -#include "models/nanbeige/nanbeige_npu.hpp" -#include "models/hunyuan/hunyuan_npu.hpp" +#include "models/llama/flm/aie2p/llama_npu.hpp" +#include "models/qwen2/flm/aie2p/qwen2_npu.hpp" +#include "models/qwen3/flm/aie2p/qwen3_npu.hpp" +#include "models/qwen2vl/flm/aie2p/qwen2vl_npu.hpp" +#include "models/qwen3vl/flm/aie2p/qwen3vl_npu.hpp" +#include "models/qwen3vl_flash/flm/aie2p/qwen3vl_flash.hpp" +#include "models/qwen3_5vl/flm/aie2p/qwen3_5vl_npu.hpp" +#include "models/qwen3_6_moe/flm/aie2p/qwen3_6_moe_npu.hpp" +#include "models/gemma/flm/aie2p/gemma_npu.hpp" +#include "models/gemma_text/flm/aie2p/gemma_text_npu.hpp" +#include "models/gemma4e/flm/aie2p/gemma4e_npu.hpp" +#include "models/gemma4e_flash/flm/aie2p/gemma4e_flash.hpp" +#include "models/gemma4_12b/flm/aie2p/gemma4_12b_npu.hpp" +#include "models/lfm2/flm/aie2p/lfm2_npu.hpp" +#include "models/phi4/flm/aie2p/phi4_npu.hpp" +#include "models/gpt_oss/flm/aie2p/gpt_oss_npu.hpp" +#include "models/nanbeige/flm/aie2p/nanbeige_npu.hpp" +#include "models/hunyuan/flm/aie2p/hunyuan_npu.hpp" #include "tokenizer/tokenizer.hpp" #include "modules/sampler.hpp" #include "utils/utils.hpp" diff --git a/src/include/AutoModel/modeling_qwen3_5_omni.hpp b/src/include/AutoModel/modeling_qwen3_5_omni.hpp index 54e744090..9b9ab064b 100644 --- a/src/include/AutoModel/modeling_qwen3_5_omni.hpp +++ b/src/include/AutoModel/modeling_qwen3_5_omni.hpp @@ -10,7 +10,7 @@ #pragma once #include "AutoModel/automodel.hpp" // base class + shared types -#include "models/qwen3_5_omni/qwen3_5_omni.hpp" +#include "models/qwen3_5_omni/flm/aie2p/qwen3_5_omni.hpp" #include "image/image_reader.hpp" #include "audio/audio_reader.hpp" diff --git a/src/include/models/gemma/gemma_npu.hpp b/src/include/models/gemma/flm/aie2p/gemma_npu.hpp similarity index 100% rename from src/include/models/gemma/gemma_npu.hpp rename to src/include/models/gemma/flm/aie2p/gemma_npu.hpp diff --git a/src/include/models/gemma/gemma_npu_sequence.hpp b/src/include/models/gemma/flm/aie2p/gemma_npu_sequence.hpp similarity index 100% rename from src/include/models/gemma/gemma_npu_sequence.hpp rename to src/include/models/gemma/flm/aie2p/gemma_npu_sequence.hpp diff --git a/src/include/models/gemma4_12b/gemma4_12b_npu.hpp b/src/include/models/gemma4_12b/flm/aie2p/gemma4_12b_npu.hpp similarity index 100% rename from src/include/models/gemma4_12b/gemma4_12b_npu.hpp rename to src/include/models/gemma4_12b/flm/aie2p/gemma4_12b_npu.hpp diff --git a/src/include/models/gemma4e/gemma4e_npu.hpp b/src/include/models/gemma4e/flm/aie2p/gemma4e_npu.hpp similarity index 100% rename from src/include/models/gemma4e/gemma4e_npu.hpp rename to src/include/models/gemma4e/flm/aie2p/gemma4e_npu.hpp diff --git a/src/include/models/gemma4e_flash/gemma4e_flash.hpp b/src/include/models/gemma4e_flash/flm/aie2p/gemma4e_flash.hpp similarity index 99% rename from src/include/models/gemma4e_flash/gemma4e_flash.hpp rename to src/include/models/gemma4e_flash/flm/aie2p/gemma4e_flash.hpp index e9ae49cad..ec273069e 100644 --- a/src/include/models/gemma4e_flash/gemma4e_flash.hpp +++ b/src/include/models/gemma4e_flash/flm/aie2p/gemma4e_flash.hpp @@ -15,7 +15,7 @@ /// them in rather than redefining them: the application builds one /// gemma4e_image_payload_t and hands it to either engine. #pragma once -#include "models/gemma4e/gemma4e_npu.hpp" +#include "models/gemma4e/flm/aie2p/gemma4e_npu.hpp" class gemma4e_flash : public causal_lm{ public: diff --git a/src/include/models/gemma_embedding/gemma_embedding.hpp b/src/include/models/gemma_embedding/flm/aie2p/gemma_embedding.hpp similarity index 100% rename from src/include/models/gemma_embedding/gemma_embedding.hpp rename to src/include/models/gemma_embedding/flm/aie2p/gemma_embedding.hpp diff --git a/src/include/models/gemma_text/gemma_text_dequant.hpp b/src/include/models/gemma_text/flm/aie2p/gemma_text_dequant.hpp similarity index 100% rename from src/include/models/gemma_text/gemma_text_dequant.hpp rename to src/include/models/gemma_text/flm/aie2p/gemma_text_dequant.hpp diff --git a/src/include/models/gemma_text/gemma_text_gemm.hpp b/src/include/models/gemma_text/flm/aie2p/gemma_text_gemm.hpp similarity index 100% rename from src/include/models/gemma_text/gemma_text_gemm.hpp rename to src/include/models/gemma_text/flm/aie2p/gemma_text_gemm.hpp diff --git a/src/include/models/gemma_text/gemma_text_lm_head.hpp b/src/include/models/gemma_text/flm/aie2p/gemma_text_lm_head.hpp similarity index 100% rename from src/include/models/gemma_text/gemma_text_lm_head.hpp rename to src/include/models/gemma_text/flm/aie2p/gemma_text_lm_head.hpp diff --git a/src/include/models/gemma_text/gemma_text_npu.hpp b/src/include/models/gemma_text/flm/aie2p/gemma_text_npu.hpp similarity index 100% rename from src/include/models/gemma_text/gemma_text_npu.hpp rename to src/include/models/gemma_text/flm/aie2p/gemma_text_npu.hpp diff --git a/src/include/models/gemma_text/gemma_text_npu_sequence.hpp b/src/include/models/gemma_text/flm/aie2p/gemma_text_npu_sequence.hpp similarity index 100% rename from src/include/models/gemma_text/gemma_text_npu_sequence.hpp rename to src/include/models/gemma_text/flm/aie2p/gemma_text_npu_sequence.hpp diff --git a/src/include/models/gpt_oss/gpt_oss_npu.hpp b/src/include/models/gpt_oss/flm/aie2p/gpt_oss_npu.hpp similarity index 100% rename from src/include/models/gpt_oss/gpt_oss_npu.hpp rename to src/include/models/gpt_oss/flm/aie2p/gpt_oss_npu.hpp diff --git a/src/include/models/gpt_oss/gpt_oss_npu_sequence.hpp b/src/include/models/gpt_oss/flm/aie2p/gpt_oss_npu_sequence.hpp similarity index 100% rename from src/include/models/gpt_oss/gpt_oss_npu_sequence.hpp rename to src/include/models/gpt_oss/flm/aie2p/gpt_oss_npu_sequence.hpp diff --git a/src/include/models/hunyuan/hunyuan_npu.hpp b/src/include/models/hunyuan/flm/aie2p/hunyuan_npu.hpp similarity index 100% rename from src/include/models/hunyuan/hunyuan_npu.hpp rename to src/include/models/hunyuan/flm/aie2p/hunyuan_npu.hpp diff --git a/src/include/models/lfm2/lfm2_npu.hpp b/src/include/models/lfm2/flm/aie2p/lfm2_npu.hpp similarity index 100% rename from src/include/models/lfm2/lfm2_npu.hpp rename to src/include/models/lfm2/flm/aie2p/lfm2_npu.hpp diff --git a/src/include/models/llama/llama_npu.hpp b/src/include/models/llama/flm/aie2p/llama_npu.hpp similarity index 100% rename from src/include/models/llama/llama_npu.hpp rename to src/include/models/llama/flm/aie2p/llama_npu.hpp diff --git a/src/include/models/llama/llama_npu_sequence.hpp b/src/include/models/llama/flm/aie2p/llama_npu_sequence.hpp similarity index 100% rename from src/include/models/llama/llama_npu_sequence.hpp rename to src/include/models/llama/flm/aie2p/llama_npu_sequence.hpp diff --git a/src/include/models/nanbeige/nanbeige_npu.hpp b/src/include/models/nanbeige/flm/aie2p/nanbeige_npu.hpp similarity index 97% rename from src/include/models/nanbeige/nanbeige_npu.hpp rename to src/include/models/nanbeige/flm/aie2p/nanbeige_npu.hpp index 2aded7118..8caf64a3e 100644 --- a/src/include/models/nanbeige/nanbeige_npu.hpp +++ b/src/include/models/nanbeige/flm/aie2p/nanbeige_npu.hpp @@ -8,7 +8,7 @@ #include "lm_config.hpp" #include "npu_utils/npu_utils.hpp" #include "tensor_utils/q4_npu_eXpress.hpp" -#include "models/nanbeige/nanbeige_npu_sequence.hpp" +#include "models/nanbeige/flm/aie2p/nanbeige_npu_sequence.hpp" #include "modules/embedding.hpp" #include "modules/lm_head.hpp" #include "modules/gemm.hpp" diff --git a/src/include/models/nanbeige/nanbeige_npu_sequence.hpp b/src/include/models/nanbeige/flm/aie2p/nanbeige_npu_sequence.hpp similarity index 100% rename from src/include/models/nanbeige/nanbeige_npu_sequence.hpp rename to src/include/models/nanbeige/flm/aie2p/nanbeige_npu_sequence.hpp diff --git a/src/include/models/phi4/phi4_npu.hpp b/src/include/models/phi4/flm/aie2p/phi4_npu.hpp similarity index 100% rename from src/include/models/phi4/phi4_npu.hpp rename to src/include/models/phi4/flm/aie2p/phi4_npu.hpp diff --git a/src/include/models/phi4/phi4_npu_sequence.hpp b/src/include/models/phi4/flm/aie2p/phi4_npu_sequence.hpp similarity index 100% rename from src/include/models/phi4/phi4_npu_sequence.hpp rename to src/include/models/phi4/flm/aie2p/phi4_npu_sequence.hpp diff --git a/src/include/models/phi4/rai/phi4_rai.hpp b/src/include/models/phi4/rai/aie_next/phi4_rai.hpp similarity index 96% rename from src/include/models/phi4/rai/phi4_rai.hpp rename to src/include/models/phi4/rai/aie_next/phi4_rai.hpp index 50903465b..6b25e7e88 100644 --- a/src/include/models/phi4/rai/phi4_rai.hpp +++ b/src/include/models/phi4/rai/aie_next/phi4_rai.hpp @@ -3,7 +3,7 @@ #include "causal_lm.hpp" #include "rai/corelib_runtime.hpp" #include "lm_config.hpp" -#include "models/phi4/rai/phi4_rai_gguf.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_gguf.hpp" #include #include diff --git a/src/include/models/phi4/rai/phi4_rai_backend.hpp b/src/include/models/phi4/rai/aie_next/phi4_rai_backend.hpp similarity index 100% rename from src/include/models/phi4/rai/phi4_rai_backend.hpp rename to src/include/models/phi4/rai/aie_next/phi4_rai_backend.hpp diff --git a/src/include/models/phi4/rai/phi4_rai_constants.hpp b/src/include/models/phi4/rai/aie_next/phi4_rai_constants.hpp similarity index 100% rename from src/include/models/phi4/rai/phi4_rai_constants.hpp rename to src/include/models/phi4/rai/aie_next/phi4_rai_constants.hpp diff --git a/src/include/models/phi4/rai/phi4_rai_gguf.hpp b/src/include/models/phi4/rai/aie_next/phi4_rai_gguf.hpp similarity index 100% rename from src/include/models/phi4/rai/phi4_rai_gguf.hpp rename to src/include/models/phi4/rai/aie_next/phi4_rai_gguf.hpp diff --git a/src/include/models/phi4/rai/phi4_rai_host.hpp b/src/include/models/phi4/rai/aie_next/phi4_rai_host.hpp similarity index 93% rename from src/include/models/phi4/rai/phi4_rai_host.hpp rename to src/include/models/phi4/rai/aie_next/phi4_rai_host.hpp index dbe7bf768..b10bd0c30 100644 --- a/src/include/models/phi4/rai/phi4_rai_host.hpp +++ b/src/include/models/phi4/rai/aie_next/phi4_rai_host.hpp @@ -1,6 +1,6 @@ #pragma once -#include "models/phi4/rai/phi4_rai_gguf.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_gguf.hpp" #include #include diff --git a/src/include/models/phi4/rai/phi4_rai_shape_plan.hpp b/src/include/models/phi4/rai/aie_next/phi4_rai_shape_plan.hpp similarity index 100% rename from src/include/models/phi4/rai/phi4_rai_shape_plan.hpp rename to src/include/models/phi4/rai/aie_next/phi4_rai_shape_plan.hpp diff --git a/src/include/models/phi4/rai/phi4_rai_weight_cache.hpp b/src/include/models/phi4/rai/aie_next/phi4_rai_weight_cache.hpp similarity index 100% rename from src/include/models/phi4/rai/phi4_rai_weight_cache.hpp rename to src/include/models/phi4/rai/aie_next/phi4_rai_weight_cache.hpp diff --git a/src/include/models/qwen2/qwen2_npu.hpp b/src/include/models/qwen2/flm/aie2p/qwen2_npu.hpp similarity index 100% rename from src/include/models/qwen2/qwen2_npu.hpp rename to src/include/models/qwen2/flm/aie2p/qwen2_npu.hpp diff --git a/src/include/models/qwen2vl/qwen2vl_npu.hpp b/src/include/models/qwen2vl/flm/aie2p/qwen2vl_npu.hpp similarity index 100% rename from src/include/models/qwen2vl/qwen2vl_npu.hpp rename to src/include/models/qwen2vl/flm/aie2p/qwen2vl_npu.hpp diff --git a/src/include/models/qwen3/qwen3_npu.hpp b/src/include/models/qwen3/flm/aie2p/qwen3_npu.hpp similarity index 100% rename from src/include/models/qwen3/qwen3_npu.hpp rename to src/include/models/qwen3/flm/aie2p/qwen3_npu.hpp diff --git a/src/include/models/qwen3/qwen3_npu_sequence.hpp b/src/include/models/qwen3/flm/aie2p/qwen3_npu_sequence.hpp similarity index 100% rename from src/include/models/qwen3/qwen3_npu_sequence.hpp rename to src/include/models/qwen3/flm/aie2p/qwen3_npu_sequence.hpp diff --git a/src/include/models/qwen3_5_omni/qwen3_5_omni.hpp b/src/include/models/qwen3_5_omni/flm/aie2p/qwen3_5_omni.hpp similarity index 100% rename from src/include/models/qwen3_5_omni/qwen3_5_omni.hpp rename to src/include/models/qwen3_5_omni/flm/aie2p/qwen3_5_omni.hpp diff --git a/src/include/models/qwen3_5vl/qwen3_5vl_npu.hpp b/src/include/models/qwen3_5vl/flm/aie2p/qwen3_5vl_npu.hpp similarity index 100% rename from src/include/models/qwen3_5vl/qwen3_5vl_npu.hpp rename to src/include/models/qwen3_5vl/flm/aie2p/qwen3_5vl_npu.hpp diff --git a/src/include/models/qwen3_6_moe/qwen3_6_moe_npu.hpp b/src/include/models/qwen3_6_moe/flm/aie2p/qwen3_6_moe_npu.hpp similarity index 100% rename from src/include/models/qwen3_6_moe/qwen3_6_moe_npu.hpp rename to src/include/models/qwen3_6_moe/flm/aie2p/qwen3_6_moe_npu.hpp diff --git a/src/include/models/qwen3vl/qwen3vl_npu.hpp b/src/include/models/qwen3vl/flm/aie2p/qwen3vl_npu.hpp similarity index 100% rename from src/include/models/qwen3vl/qwen3vl_npu.hpp rename to src/include/models/qwen3vl/flm/aie2p/qwen3vl_npu.hpp diff --git a/src/include/models/qwen3vl_flash/qwen3vl_flash.hpp b/src/include/models/qwen3vl_flash/flm/aie2p/qwen3vl_flash.hpp similarity index 97% rename from src/include/models/qwen3vl_flash/qwen3vl_flash.hpp rename to src/include/models/qwen3vl_flash/flm/aie2p/qwen3vl_flash.hpp index 75dce8ed0..10ac1f22a 100644 --- a/src/include/models/qwen3vl_flash/qwen3vl_flash.hpp +++ b/src/include/models/qwen3vl_flash/flm/aie2p/qwen3vl_flash.hpp @@ -17,7 +17,7 @@ /// the application builds one qwen3vl_image_payload_t and hands it to either /// engine. #pragma once -#include "models/qwen3vl/qwen3vl_npu.hpp" +#include "models/qwen3vl/flm/aie2p/qwen3vl_npu.hpp" class qwen3vl_flash : public causal_lm{ diff --git a/src/src/main.cpp b/src/src/main.cpp index e32fca6ab..50282c1af 100644 --- a/src/src/main.cpp +++ b/src/src/main.cpp @@ -576,6 +576,10 @@ int main(int argc, char* argv[]) { // Set when corelib failed but a direct device was opened anyway: the rai // backend is gone, the rest of the build is not. std::string corelib_note; + + + header_print("DEBUG", "RAI enabled!"); + try { const auto runtime = flm::corelib::CorelibRuntime::GetOrCreate(std::filesystem::path(exe_dir)); @@ -588,15 +592,16 @@ int main(int argc, char* argv[]) { if (npu_device == nullptr) { npu_open_error = "corelib started but reports no NPU device on this machine"; - DO_VERBOSE(1, { - header_print("FLM", "corelib reports no NPU device on this machine"); - }); + header_print("DEBUG", "corelib reports no NPU device on this machine"); + } + else { + header_print("DEBUG", "got device from corelib"); } } catch (const std::exception& e) { // A box with no NPU must still run `flm list`/`pull`/`version`, so this // stays a null device rather than an error, as in the non-rai path. npu_open_error = std::string("corelib unavailable: ") + e.what(); - DO_VERBOSE(1, { header_print("FLM", "corelib unavailable: " << e.what()); }); + header_print("FLM", "corelib unavailable: " << e.what()); } if (npu_device == nullptr) { // corelib having no device is a reason for the rai backend to be @@ -642,9 +647,6 @@ int main(int argc, char* argv[]) { const bool needs_npu = parsed_args.command == "run" || parsed_args.command == "serve" || parsed_args.command == "bench" || parsed_args.command == "validate"; - if (print_status && needs_npu) { - header_print("FLM", "NPU platform: " << utils::platform_id(platform)); - } // Say once, here, why there is no device. The commands below cannot run // without one, and the message they eventually produce names the symptom diff --git a/src/test/gemma4_12b/Makefile b/src/test/gemma4_12b/Makefile index 6077f94a5..522cf5c5a 100644 --- a/src/test/gemma4_12b/Makefile +++ b/src/test/gemma4_12b/Makefile @@ -30,7 +30,7 @@ SOURCES += ../../common/audio/audio_reader.cpp SOURCES += ../../common/tokenizer/tokenizer.cpp SOURCES += ../../common/modules/sampler.cpp -HEADERS += ../../include/models/gemma4_12b/gemma4_12b_npu.hpp +HEADERS += ../../include/models/gemma4_12b/flm/aie2p/gemma4_12b_npu.hpp HEADERS += ../../include/AutoModel/modeling_gemma4_12b.hpp ifeq ($(WSL), 0) diff --git a/src/test/gemma4e_flash/Makefile b/src/test/gemma4e_flash/Makefile index ae3a726ca..a67fd2f10 100644 --- a/src/test/gemma4e_flash/Makefile +++ b/src/test/gemma4e_flash/Makefile @@ -30,8 +30,8 @@ SOURCES += ../../common/audio/audio_reader.cpp SOURCES += ../../common/tokenizer/tokenizer.cpp SOURCES += ../../common/modules/sampler.cpp -HEADERS += ../../include/models/gemma4e_flash/gemma4e_flash.hpp -HEADERS += ../../include/models/gemma4e/gemma4e_npu.hpp +HEADERS += ../../include/models/gemma4e_flash/flm/aie2p/gemma4e_flash.hpp +HEADERS += ../../include/models/gemma4e/flm/aie2p/gemma4e_npu.hpp # Test task selection; text mode is default # 0: text diff --git a/src/test/hunyuan_npu/Makefile b/src/test/hunyuan_npu/Makefile index debdb74e7..5698a195f 100644 --- a/src/test/hunyuan_npu/Makefile +++ b/src/test/hunyuan_npu/Makefile @@ -41,7 +41,7 @@ SOURCES += ../../common/AutoModel/modeling_hunyuan.cpp SOURCES += ../../common/tokenizer/tokenizer.cpp SOURCES += ../../common/modules/sampler.cpp -HEADERS += ../../include/models/hunyuan/hunyuan_npu.hpp +HEADERS += ../../include/models/hunyuan/flm/aie2p/hunyuan_npu.hpp HEADERS += ../../include/AutoModel/modeling_hunyuan.hpp HEADERS += bench_samples.hpp diff --git a/src/test/phi4_rai/CMakeLists.txt b/src/test/phi4_rai/CMakeLists.txt index 8b62fb819..9a84ae539 100644 --- a/src/test/phi4_rai/CMakeLists.txt +++ b/src/test/phi4_rai/CMakeLists.txt @@ -71,7 +71,7 @@ target_compile_definitions(test_real_corelib PRIVATE RYZENAI_CORELIB_STATIC=1) add_executable(test_phi4_gguf test_phi4_gguf.cpp fake_corelib.cpp "${FLM_SOURCE_DIR}/common/rai/corelib_api.cpp" - "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_gguf.cpp") + "${FLM_SOURCE_DIR}/common/models/phi4/rai/aie_next/phi4_rai_gguf.cpp") target_include_directories(test_phi4_gguf PRIVATE "${CMAKE_CURRENT_LIST_DIR}" "${FLM_SOURCE_DIR}/include" @@ -80,7 +80,7 @@ target_compile_definitions(test_phi4_gguf PRIVATE RYZENAI_CORELIB_STATIC=1) add_executable(test_phi4_host test_phi4_host.cpp - "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_host.cpp") + "${FLM_SOURCE_DIR}/common/models/phi4/rai/aie_next/phi4_rai_host.cpp") target_include_directories(test_phi4_host PRIVATE "${CMAKE_CURRENT_LIST_DIR}" "${FLM_SOURCE_DIR}/include") @@ -88,7 +88,7 @@ target_include_directories(test_phi4_host PRIVATE add_executable(test_phi4_shape_plan test_phi4_shape_plan.cpp fake_corelib.cpp "${FLM_SOURCE_DIR}/common/rai/corelib_api.cpp" - "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_shape_plan.cpp") + "${FLM_SOURCE_DIR}/common/models/phi4/rai/aie_next/phi4_rai_shape_plan.cpp") target_include_directories(test_phi4_shape_plan PRIVATE "${CMAKE_CURRENT_LIST_DIR}" "${FLM_SOURCE_DIR}/include" @@ -99,11 +99,11 @@ add_executable(test_phi4_engine test_phi4_engine.cpp fake_corelib.cpp "${FLM_SOURCE_DIR}/common/rai/corelib_api.cpp" "${FLM_SOURCE_DIR}/common/rai/corelib_runtime.cpp" - "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_gguf.cpp" - "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_host.cpp" - "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_shape_plan.cpp" - "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai.cpp" - "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_weight_cache.cpp") + "${FLM_SOURCE_DIR}/common/models/phi4/rai/aie_next/phi4_rai_gguf.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/rai/aie_next/phi4_rai_host.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/rai/aie_next/phi4_rai_shape_plan.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/rai/aie_next/phi4_rai.cpp" + "${FLM_SOURCE_DIR}/common/models/phi4/rai/aie_next/phi4_rai_weight_cache.cpp") target_include_directories(test_phi4_engine PRIVATE "${CMAKE_CURRENT_LIST_DIR}" "${FLM_SOURCE_DIR}/include" @@ -130,7 +130,7 @@ add_executable(test_phi4_frontend test_phi4_frontend.cpp fake_corelib.cpp ${PHI4_FRONTEND_SOURCES} ${CORELIB_SOURCES} - "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_gguf.cpp") + "${FLM_SOURCE_DIR}/common/models/phi4/rai/aie_next/phi4_rai_gguf.cpp") target_include_directories(test_phi4_frontend PRIVATE "${CMAKE_CURRENT_LIST_DIR}" "${FLM_SOURCE_DIR}/include" @@ -239,7 +239,7 @@ function(add_frontend_compile_guard TARGET_NAME ENABLE_CORELIB) $<$:/wd4005 /wd4244>) if(ENABLE_CORELIB) target_sources(${TARGET_NAME} PRIVATE - "${FLM_SOURCE_DIR}/common/models/phi4/rai/phi4_rai_backend.cpp") + "${FLM_SOURCE_DIR}/common/models/phi4/rai/aie_next/phi4_rai_backend.cpp") target_compile_definitions(${TARGET_NAME} PRIVATE FLM_ENABLE_RAI=1) target_include_directories(${TARGET_NAME} PRIVATE diff --git a/src/test/phi4_rai/test_phi4_engine.cpp b/src/test/phi4_rai/test_phi4_engine.cpp index 55bc3cb7b..d87e035c0 100644 --- a/src/test/phi4_rai/test_phi4_engine.cpp +++ b/src/test/phi4_rai/test_phi4_engine.cpp @@ -1,7 +1,7 @@ -#include "models/phi4/rai/phi4_rai.hpp" -#include "models/phi4/rai/phi4_rai_constants.hpp" -#include "models/phi4/rai/phi4_rai_host.hpp" -#include "models/phi4/rai/phi4_rai_weight_cache.hpp" +#include "models/phi4/rai/aie_next/phi4_rai.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_constants.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_host.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_weight_cache.hpp" #include #include #include diff --git a/src/test/phi4_rai/test_phi4_frontend.cpp b/src/test/phi4_rai/test_phi4_frontend.cpp index 81156d484..88edfaf92 100644 --- a/src/test/phi4_rai/test_phi4_frontend.cpp +++ b/src/test/phi4_rai/test_phi4_frontend.cpp @@ -9,8 +9,8 @@ #include #if defined(FLM_ENABLE_RAI) #include -#include -#include +#include +#include #endif #include "server.hpp" diff --git a/src/test/phi4_rai/test_phi4_gguf.cpp b/src/test/phi4_rai/test_phi4_gguf.cpp index 9f15f4e09..42aebdbc0 100644 --- a/src/test/phi4_rai/test_phi4_gguf.cpp +++ b/src/test/phi4_rai/test_phi4_gguf.cpp @@ -1,8 +1,8 @@ #include "gguf_fixture.hpp" #include "fake_corelib.hpp" #include "rai/corelib_api.hpp" -#include "models/phi4/rai/phi4_rai_constants.hpp" -#include "models/phi4/rai/phi4_rai_gguf.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_constants.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_gguf.hpp" #include "test_support.hpp" #include diff --git a/src/test/phi4_rai/test_phi4_host.cpp b/src/test/phi4_rai/test_phi4_host.cpp index 862875686..99b90738e 100644 --- a/src/test/phi4_rai/test_phi4_host.cpp +++ b/src/test/phi4_rai/test_phi4_host.cpp @@ -1,4 +1,4 @@ -#include "models/phi4/rai/phi4_rai_host.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_host.hpp" #include "test_support.hpp" #include diff --git a/src/test/phi4_rai/test_phi4_shape_plan.cpp b/src/test/phi4_rai/test_phi4_shape_plan.cpp index 75b5ac343..06909a4ea 100644 --- a/src/test/phi4_rai/test_phi4_shape_plan.cpp +++ b/src/test/phi4_rai/test_phi4_shape_plan.cpp @@ -1,5 +1,5 @@ -#include "models/phi4/rai/phi4_rai_shape_plan.hpp" -#include "models/phi4/rai/phi4_rai_constants.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_shape_plan.hpp" +#include "models/phi4/rai/aie_next/phi4_rai_constants.hpp" #include "fake_corelib.hpp" #include "test_support.hpp" diff --git a/src/test/qwen3_5_omni_npu/Makefile b/src/test/qwen3_5_omni_npu/Makefile index c4f44ae35..1453e2764 100644 --- a/src/test/qwen3_5_omni_npu/Makefile +++ b/src/test/qwen3_5_omni_npu/Makefile @@ -32,7 +32,7 @@ SOURCES += ../../common/audio/audio_reader.cpp SOURCES += ../../common/tokenizer/tokenizer.cpp SOURCES += ../../common/modules/sampler.cpp -HEADERS += ../../include/models/qwen3_5_omni/qwen3_5_omni.hpp +HEADERS += ../../include/models/qwen3_5_omni/flm/aie2p/qwen3_5_omni.hpp HEADERS += ../../include/AutoModel/modeling_qwen3_5_omni.hpp # omni mode is default; diff --git a/src/test/qwen3vl_flash/Makefile b/src/test/qwen3vl_flash/Makefile index d4553a8d4..faf9f7ebd 100644 --- a/src/test/qwen3vl_flash/Makefile +++ b/src/test/qwen3vl_flash/Makefile @@ -26,8 +26,8 @@ SOURCES += ../../common/image/image_reader.cpp SOURCES += ../../common/tokenizer/tokenizer.cpp SOURCES += ../../common/modules/sampler.cpp -HEADERS += ../../include/models/qwen3vl_flash/qwen3vl_flash.hpp -HEADERS += ../../include/models/qwen3vl/qwen3vl_npu.hpp +HEADERS += ../../include/models/qwen3vl_flash/flm/aie2p/qwen3vl_flash.hpp +HEADERS += ../../include/models/qwen3vl/flm/aie2p/qwen3vl_npu.hpp ifeq ($(WSL), 0) From cb8a19ddd406826d46209512a584efa77f435f4c Mon Sep 17 00:00:00 2001 From: alfxu_amdeng Date: Thu, 24 Sep 2026 21:36:19 -0700 Subject: [PATCH 08/17] fix(rai): bind the rotary tables as host views flat_mha reads the rotary tables on the host to build the table the kernel binds; they never reach the device, and corelib rejects a device tensor in those slots. They were being written as device tensors, which 0.5.0 no longer accepts. create_host_view was missing from the resolved symbol table entirely, so add it, along with a UniqueHostView alias for the lifetime it implies: a host view borrows the caller's bytes rather than copying them, so RopeTables is now a member declared ahead of the views that borrow it -- members are destroyed in reverse declaration order, so the tables outlive the views. Co-Authored-By: Claude Opus 5 --- .../models/phi4/rai/aie_next/phi4_rai.cpp | 27 ++++++++++++++----- src/include/rai/corelib_api.hpp | 1 + src/include/rai/corelib_object.hpp | 4 +++ 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/src/common/models/phi4/rai/aie_next/phi4_rai.cpp b/src/common/models/phi4/rai/aie_next/phi4_rai.cpp index 87f1e5e18..ba20d7b85 100644 --- a/src/common/models/phi4/rai/aie_next/phi4_rai.cpp +++ b/src/common/models/phi4/rai/aie_next/phi4_rai.cpp @@ -90,7 +90,16 @@ struct phi4_rai::Impl { std::array q_weights, k_weights, v_weights, o_weights; std::array mlp_weights; UniqueMatMulWeights lm_weights; - UniqueTensor hidden, residual, skip, q, k, attention, lm_input, logits, cosine, sine; + UniqueTensor hidden, residual, skip, q, k, attention, lm_input, logits; + /// \brief the rotary tables, kept for the life of the engine + /// \note The host views below borrow these bytes rather than copying them, + /// so this must outlive them. Declared first on purpose: members are + /// destroyed in reverse declaration order. + RopeTables rope; + /// \note flat_mha reads the rotary tables on the host to build the table + /// the kernel binds; they never reach the device. corelib takes them + /// as host views and rejects a device tensor in these slots. + UniqueHostView cosine, sine; std::array k_cache, v_cache; TensorView embedding; FloatTensorView first_norm_scale; @@ -125,7 +134,7 @@ struct phi4_rai::Impl { std::optional factors; try { factors=package->RequireF32("rope_factors_short.weight",std::array{48}); } catch (const std::runtime_error&) {} - auto rope=BuildShortRopeTables(package->Metadata(),factors); + rope=BuildShortRopeTables(package->Metadata(),factors); auto final_bf=ConvertF32ToBf16(final_norm.values); std::array,kLayerCount> an_bf,fn_bf; for(std::size_t i=0;iCheck(api->functions().tensor_write(cosine.get(),ryzenai_corelib_data_type_fp32,rope.cosine.data(),rope.cosine.size(),0),"ryzenai_corelib_tensor_write cosine"); - api->Check(api->functions().tensor_write(sine.get(),ryzenai_corelib_data_type_fp32,rope.sine.data(),rope.sine.size(),0),"ryzenai_corelib_tensor_write sine"); + auto host_view=[&](const std::vector& values,const char* label){ + std::array shape{kMaxSequenceLength,48};void* p=nullptr; + api->Check(api->functions().create_host_view(ryzenai_corelib_data_type_fp32,shape.data(),shape.size(),values.data(),&p),std::string("ryzenai_corelib_create_host_view ")+label); + return UniqueHostView(api,p); + }; + cosine=host_view(rope.cosine,"cosine"); + sine=host_view(rope.sine,"sine"); phases.device_tensors = phases.Lap(); phases.Report(); } @@ -439,7 +451,8 @@ struct phi4_rai::Impl { api->Check(api->functions().matmul(stream.get(),hidden_kv.get(),v_weights[i].get(),win.get()),"ryzenai_corelib_matmul_bf16 value layer "+std::to_string(i)); api->Check(api->functions().flat_mha(stream.get(),&plan.attention_desc(),q_mha.get(),k_mha.get(),position,cosine.get(),sine.get(),k_cache[i].get(),v_cache[i].get(),attention_mha.get()),"ryzenai_corelib_flat_mha_bf16 layer "+std::to_string(i)); api->Check(api->functions().matmul(stream.get(),attention_mm.get(),o_weights[i].get(),hidden_out.get()),"ryzenai_corelib_matmul_bf16 output layer "+std::to_string(i)); - api->Check(api->functions().ssmlp(stream.get(),hidden_mlp.get(),res,mlp_weights[i].get(),sk,hidden_mlp.get()),"ryzenai_corelib_ssmlp_bf16 layer "+std::to_string(i));std::swap(res,sk); + api->Check(api->functions().ssmlp(stream.get(),hidden_mlp.get(),res,mlp_weights[i].get(),sk,hidden_mlp.get()),"ryzenai_corelib_ssmlp_bf16 layer "+std::to_string(i)); + std::swap(res,sk); } api->Check(api->functions().stream_synchronize(stream.get()),"ryzenai_corelib_stream_synchronize hidden"); std::vector row(kHiddenSize);api->Check(api->functions().tensor_read(hidden.get(),ryzenai_corelib_data_type_bf16,row.data(),row.size(),(ids.size()-1)*kHiddenSize),"ryzenai_corelib_tensor_read final hidden row"); diff --git a/src/include/rai/corelib_api.hpp b/src/include/rai/corelib_api.hpp index 16f5c6edd..50315e523 100644 --- a/src/include/rai/corelib_api.hpp +++ b/src/include/rai/corelib_api.hpp @@ -28,6 +28,7 @@ X(stream_synchronize, ryzenai_corelib_stream_synchronize) \ X(create_device_tensor, ryzenai_corelib_create_device_tensor) \ X(create_tensor_window, ryzenai_corelib_create_tensor_window) \ + X(create_host_view, ryzenai_corelib_create_host_view) \ X(tensor_write, ryzenai_corelib_tensor_write) \ X(tensor_read, ryzenai_corelib_tensor_read) \ X(tensor_get_byte_size, ryzenai_corelib_tensor_get_byte_size) \ diff --git a/src/include/rai/corelib_object.hpp b/src/include/rai/corelib_object.hpp index a8d92a109..19d60bd2d 100644 --- a/src/include/rai/corelib_object.hpp +++ b/src/include/rai/corelib_object.hpp @@ -12,6 +12,7 @@ struct TensorTag {}; struct TensorWindowTag {}; struct MatMulWeightsTag {}; struct SsMlpWeightsTag {}; +struct HostViewTag {}; template class UniqueObject final { @@ -58,5 +59,8 @@ using UniqueTensor = UniqueObject; using UniqueTensorWindow = UniqueObject; using UniqueMatMulWeights = UniqueObject; using UniqueSsMlpWeights = UniqueObject; +/// \note A host view borrows the caller's bytes and never copies them, so +/// whatever owns those bytes must outlive the view. +using UniqueHostView = UniqueObject; } // namespace flm::corelib From 0fed8a9267a4252acf60eda2775885c6f00ff32c Mon Sep 17 00:00:00 2001 From: alfxu_amdeng Date: Thu, 24 Sep 2026 21:37:12 -0700 Subject: [PATCH 09/17] fix(phi4): say why inference failed instead of swallowing the reason Both catch sites discarded the exception and threw a fixed message, so every backend failure arrived as "Inference failed" with nothing to act on -- the corelib label naming the failing op was lost at the catch. Route both through one fail_inference() that appends the backend's own detail. It also distinguishes the two recovery paths: a poisoned backend needs an unload/reload, an unpoisoned one only lost the conversation, and the message now says which. Co-Authored-By: Claude Opus 5 --- src/common/AutoModel/modeling_phi4.cpp | 12 +++++++++--- src/include/AutoModel/modeling_phi4.hpp | 5 ++++- 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/src/common/AutoModel/modeling_phi4.cpp b/src/common/AutoModel/modeling_phi4.cpp index eb8849cdd..021a3062e 100644 --- a/src/common/AutoModel/modeling_phi4.cpp +++ b/src/common/AutoModel/modeling_phi4.cpp @@ -100,12 +100,14 @@ std::string Phi4::apply_chat_template(nlohmann::ordered_json& messages, return chat_tmpl->apply(inputs); } -void Phi4::fail_inference() { +void Phi4::fail_inference(const std::string& detail) { const bool poisoned = backend_ && backend_->poisoned(); _shared_after_inference_failure(poisoned); - throw ModelRequestError(500, true, poisoned + std::string message = poisoned ? "Inference failed; unload/reload is required because the model is poisoned" - : "Inference failed; the current conversation was cleared"); + : "Inference failed; the current conversation was cleared"; + if (!detail.empty()) message += " (" + detail + ")"; + throw ModelRequestError(500, true, message); } bool Phi4::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, @@ -133,6 +135,8 @@ bool Phi4::insert(chat_meta_info_t& meta_info, lm_uniform_input_t& input, 0, input.requested_max_new_tokens); } catch (const ModelRequestError&) { throw; + } catch (const std::exception& error) { + fail_inference(error.what()); } catch (...) { fail_inference(); } @@ -146,6 +150,8 @@ std::string Phi4::generate(chat_meta_info_t& meta_info, int length_limit, return _shared_generate(meta_info, length_limit, os, std::move(is_cancelled)); } catch (const ModelRequestError&) { throw; + } catch (const std::exception& error) { + fail_inference(error.what()); } catch (...) { fail_inference(); } diff --git a/src/include/AutoModel/modeling_phi4.hpp b/src/include/AutoModel/modeling_phi4.hpp index 56d15499b..032ed3d34 100644 --- a/src/include/AutoModel/modeling_phi4.hpp +++ b/src/include/AutoModel/modeling_phi4.hpp @@ -30,7 +30,10 @@ class Phi4 : public AutoModel { /// \brief Turn a failed inference into a request error, clearing the session /// \throws ModelRequestError 500, always - [[noreturn]] void fail_inference(); + /// \param detail the backend's own message, when the failure carried one + /// \note The detail is appended to the thrown message. Without it the + /// corelib label that names the failing op is lost at the catch. + [[noreturn]] void fail_inference(const std::string& detail = {}); public: explicit Phi4(flm_rt::device* npu_device_inst); From 2bf92e8fe4c57fb2d5e2bd297397892e60f7423a Mon Sep 17 00:00:00 2001 From: alfxu_amdeng Date: Thu, 24 Sep 2026 21:37:12 -0700 Subject: [PATCH 10/17] fix(rai): do not gate a corelib build on the stock NPU stack checks The checks read the NPU's geometry and this process's memlock limit and hold them against what FastFlowLM's own kernel flow needs. A corelib build does not use that flow -- corelib owns device setup and brings the NPU up itself -- so the checks answer a question this build does not ask, against a threshold describing kernels it does not run. Left in, they fail a working rai install on hardware corelib runs on perfectly well. The JSON keeps the shape callers already parse but reports the checks as skipped rather than claiming they passed. Also drops three DEBUG prints left over from bringing the backend up. Co-Authored-By: Claude Opus 5 --- src/src/main.cpp | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/src/src/main.cpp b/src/src/main.cpp index 50282c1af..e15aceba1 100644 --- a/src/src/main.cpp +++ b/src/src/main.cpp @@ -216,6 +216,38 @@ std::string identify_npu_arch() { static bool sanity_check_npu_stack(bool quiet, bool json_output = false) { bool print_human = !quiet && !json_output; +#ifdef FLM_ENABLE_RAI + // Everything below reads the NPU's geometry and this process's memlock + // limit from the driver and holds them against what FastFlowLM's own kernel + // flow needs. A corelib build does not use that flow. corelib owns device + // setup and brings the NPU up itself -- main() already reports whether that + // worked, and says why when it did not -- so these checks answer a question + // this build does not ask, against a threshold that describes FastFlowLM's + // own kernels rather than the parts corelib supports. Left in, they fail a + // working rai install on hardware corelib runs on perfectly well. + // + // The JSON keeps the shape callers already parse, but reports the checks as + // skipped rather than claiming they passed: "ready" here means "this build + // does not gate on the stock stack". A caller that wants to know whether the + // NPU actually came up should look at whether flm opened a device. + if (json_output) { + nlohmann::json validation_json = { + {"object", "npu_stack_validation"}, +#ifdef _WIN32 + {"platform", "windows"}, +#else + {"platform", "linux"}, +#endif + {"backend", "rai"}, + {"checks_skipped", true}, + {"ready", true} + }; + std::cout << validation_json.dump(4) << std::endl; + } else if (print_human) { + header_print("FLM", "corelib backend: skipping NPU stack checks (corelib owns device setup)"); + } + return true; +#else #ifndef _WIN32 nlohmann::json validation_json = { {"object", "npu_stack_validation"}, @@ -457,6 +489,7 @@ static bool sanity_check_npu_stack(bool quiet, bool json_output = false) { } return true; #endif +#endif // FLM_ENABLE_RAI } @@ -609,7 +642,7 @@ int main(int argc, char* argv[]) { // family is served by the flm backend, which opens its own device and // never touches corelib -- so opening one directly keeps those models // working instead of failing the whole process over a backend they do - // not use. On a box whose NPU has no creatable AIE4 hw_context this is + // not use. On a box whose NPU has no creatable AIE_NEXT hw_context this is // the difference between "rai is unavailable" and "flm has no NPU". // // This does not bring back the two-device bug, where a buffer created From 8d8bb277c451b0d7be859f3b810fdd4261657cf4 Mon Sep 17 00:00:00 2001 From: alfxu_amdeng Date: Thu, 24 Sep 2026 21:38:01 -0700 Subject: [PATCH 11/17] docs(rai): add a Linux setup guide, and redact a stack identifier The guide covers a from-scratch build: the corelib/DynamicDispatch prefix and the exact version the headers are pinned to, how the corelib lookup resolves, the XRT library the installer stages over the real one, the memlock limit and why a limits file does not apply to an already-running shell, and the known corelib stream-reuse bug that currently blocks the backend. home_install.sh's comments named an internal stack identifier that should not appear in output; use the neutral name instead. Co-Authored-By: Claude Opus 5 --- docs/rai-setup-linux.md | 248 ++++++++++++++++++++++++++++++++++++++++ src/home_install.sh | 8 +- 2 files changed, 252 insertions(+), 4 deletions(-) create mode 100644 docs/rai-setup-linux.md diff --git a/docs/rai-setup-linux.md b/docs/rai-setup-linux.md new file mode 100644 index 000000000..a81527e2f --- /dev/null +++ b/docs/rai-setup-linux.md @@ -0,0 +1,248 @@ +# Building the `--rai` backend on Linux from scratch + +How to get from a bare checkout to `flm run phi4-mini-it-rai` on a machine with +an AMD NPU. The `--rai` backend reaches kernels through **ryzenai-corelib** +instead of FastFlowLM's own kernel flow, and corelib lives outside this +repository, so most of the work is standing up that prefix. + +Written against the working install on this machine. Paths use `$P` for the +dependency prefix (here `/home/alfxu/ddprefix`); substitute your own. + +> **Status:** the backend builds, loads a model and dispatches, but a corelib +> bug makes it very slow — see [Known issues](#known-issues) before you start, +> so the performance is not a surprise. + +--- + +## 0. What you need first + +| Thing | This machine | Check | +|---|---|---| +| AMD NPU + `amdxdna` driver | `/dev/accel/accel0` | `ls /dev/accel/` | +| XRT | `/opt/xilinx/xrt` | `source /opt/xilinx/xrt/setup.sh && xrt-smi examine` | +| CMake ≥ 3.21, Ninja, GCC with C++17 | GCC 15 | `cmake --version` | +| Raised `memlock` limit | see step 4 | `ulimit -l` | + +No sudo is needed for anything below **except** the `memlock` limits file in +step 4 and installing XRT itself. + +--- + +## 1. Dependency prefix: DynamicDispatch, then corelib + +This is the hard part and it is already written up in **`$P/BUILD-NOTES.md`** — +read that first. It records the two things that actually matter (the required +DD branch, and `BUILD_SHARED_LIBS=ON` for DD), the sudo-free dependency prefix, +the non-PIC link traps, and the exact `cmake` command lines for both projects. + +Do not re-derive it. The short version: + +```bash +P=/home/alfxu/ddprefix # your dependency prefix + +# DynamicDispatch -> $P/install-dds (see BUILD-NOTES.md for the full line) +cmake -S DynamicDispatch -B build-dds ... -DCMAKE_INSTALL_PREFIX=$P/install-dds +cmake --build build-dds -j"$(nproc)" --target install + +# ryzenai-corelib -> $P/install-corelib +cmake -S ryzenai-corelib -B build-corelib -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_INSTALL_PREFIX=$P/install-corelib \ + -DRYZENAI_CORELIB_STAGE_RUNTIME_DLLS=OFF \ + -DCMAKE_PREFIX_PATH="$P/install-dds;$P/shim;$P/root/usr;$P/root/usr/lib/x86_64-linux-gnu;/opt/xilinx/xrt" +cmake --build build-corelib -j"$(nproc)" +``` + +**Version must match exactly.** `src/include/rai/corelib_api.hpp` has a +`#error` for anything other than corelib **0.5.0**: + +```bash +grep -E 'CORELIB_VERSION_(MAJOR|MINOR|PATCH)' \ + $P/install-corelib/include/ryzenai/corelib.h +``` + +Must print `0`, `5`, `0`. (`BUILD-NOTES.md` still says 0.4.0 in its opening +line — that text is stale; the installed headers are 0.5.0.) + +### Verify corelib before going further + +```bash +source /opt/xilinx/xrt/setup.sh +export LD_LIBRARY_PATH=$P/install-dds/lib:$P/install-corelib/lib:/opt/xilinx/xrt/lib:$LD_LIBRARY_PATH +$P/build-corelib/tests/ryzenai_corelib_tests +``` + +All tests should pass. Do this now — if corelib cannot reach the NPU, every +failure later in the FastFlowLM build will be reported as something else. + +Note the suite only dispatches **one** op on hardware, **once**. A pass means +device setup and the build are sound; it does not exercise repeated dispatch. + +--- + +## 2. Build FastFlowLM with `--rai` + +The installer does configure, build and relocate in one step: + +```bash +cd /home/alfxu/FastFlowLM/src +RYZENAI_CORELIB_ROOT=$P/install-corelib ./home_install.sh --rai +``` + +`RYZENAI_CORELIB_ROOT` is needed **once** — CMake caches it. If you ever delete +`src/build/`, you must pass it again, because the cache goes with it. + +What the flags do: + +- `--rai` selects the `linux-rai-on` preset (`-DFLM_ENABLE_RAI=ON`) and installs + to `/scratch/$USER/flm_exe_rai` rather than `~/flm_exe`. A rai prefix stages + corelib and DynamicDispatch, which are far too large for a home directory, and + a rai build and a stock one would otherwise overwrite each other. +- `FLM_PREFIX=/path` overrides the install location. +- `--no-build` installs an existing `build/` tree without rebuilding. + +### How the corelib lookup resolves + +`CMakeLists.txt:57-85`, in order: + +| | Headers (`ryzenai/corelib.h`) | Library (`libryzenai_corelib`) | +|---|---|---| +| 1 | `src/include/` (vendored) | `src/lib/` (vendored) | +| 2 | `$RYZENAI_CORELIB_ROOT/include` | `$RYZENAI_CORELIB_ROOT/lib` | +| 3 | `CMAKE_PREFIX_PATH` | `CMAKE_PREFIX_PATH` | + +So dropping the headers into `src/include/` and `libryzenai_corelib.so` into +`src/lib/` is an alternative to the variable entirely. + +**If you put a `.so` in `src/lib/`, it must keep its RUNPATH.** Copy the one +from the corelib *build tree*, not a stripped or patched copy: + +```bash +objdump -p src/lib/libryzenai_corelib.so | grep RUNPATH +# -> /home/alfxu/ddprefix/install-dds/lib:/opt/xilinx/xrt/lib: +``` + +Without that RUNPATH it will not find DynamicDispatch at run time. + +### Rebuilding after a source change + +```bash +cd /home/alfxu/FastFlowLM/src +cmake --build build -j"$(nproc)" +cp build/flm /scratch/$USER/flm_exe_rai/flm +``` + +Faster than re-running the installer, and it skips the XRT-shadowing problem in +step 3. + +--- + +## 3. Fix the shadowed XRT library — every time you run the installer + +`home_install.sh` stages `libxrt_coreutil.so.2` into the install prefix, where +it shadows the real XRT and breaks the NPU with confusing errors. Delete it +after **every** installer run: + +```bash +rm -f /scratch/$USER/flm_exe_rai/lib/libxrt_coreutil.so.2 +``` + +This is a known bug in the installer's `ldd` staging walk +(`home_install.sh:251-261`), not yet fixed. A rai prefix deliberately bundles no +XRT — `flm_env.sh` pins `XILINX_XRT` to the real root instead — so the staged +copy is wrong by construction. + +--- + +## 4. Raise the `memlock` limit + +A Phi-4 model pins several GB. The default 8 MB fails at load with a pinning +error that does not name the limit. + +```bash +# /etc/security/limits.d/99-memlock.conf (needs sudo, once) +alfxu hard memlock 16777216 +``` + +That is the **hard** limit in KB (16 GiB). Then, in the shell that runs `flm`: + +```bash +ulimit -l unlimited # or: ulimit -l 16777216 +ulimit -l # confirm +``` + +**The limits file applies only to sessions started after it was written.** A +long-running shell, a tmux server, or any process whose chain predates the file +keeps the old 8 MB cap and cannot raise it. If `ulimit -l` still reports `8192`, +log out and back in, and `tmux kill-server` if you use tmux. + +--- + +## 5. Run + +```bash +source /opt/xilinx/xrt/setup.sh +source /scratch/$USER/flm_exe_rai/flm_env.sh +/scratch/$USER/flm_exe_rai/flm run phi4-mini-it-rai -c 4096 +``` + +`flm_env.sh` sets `FLM_CONFIG_PATH`, `FLM_XCLBIN_PATH`, `FLM_MODEL_PATH` +(`/scratch/alfxu`) and pins `XILINX_XRT`. It does **not** set `ulimit` — step 4 +is on you. + +Models live under `$FLM_MODEL_PATH/`; `phi4-mini-it-rai` is the one set up +here, and it must also have an entry in +`share/flm/model_list.json` for `flm run` to resolve the name. + +--- + +## Known issues + +### Stream reuse — the big one + +A corelib stream faults on a later dispatch: the array raises stream switch port +parity errors, the driver tears the hardware context down, and XRT reports +`ERT_CMD_STATE_TIMEOUT` against whichever op was in flight. It is not a +FastFlowLM bug — there is a ~90-line standalone reproducer, with a full write-up +and a report note, in **`~/corelib-stream-repro/`**. + +FastFlowLM works around it on Linux by synchronizing after every dispatch and +replacing the stream, guarded by `#if defined(__linux__)` in +`src/common/models/phi4/rai/aie_next/phi4_rai.cpp`. That costs all the +pipelining: expect roughly **1.6 s per token**, under 1 tok/s. The workaround is +annotated for removal once corelib no longer needs it. + +### Misleading diagnostics + +When a dispatch fails, ignore these — they are printed unconditionally on that +XRT error path and describe nothing about real device state: + +- `CTX_STATUS_UNASSIGNED` +- `ctx_error_type = NPU_ASYNC_EVENT_CTX_ERR_HWSCH_FAILURE` +- `number of uC reported = 0` +- the "N dispatch(es) outstanding" count — it is the size of a list that is + never pruned, not a failure count +- `amdxdna_ubuf_get_pages: Failed to pin pages ret -14` in `dmesg` — a red + herring alongside the stream fault, confirmed by testing with + `RYZENAI_CORELIB_SAFE_MODE=1`, which bypasses host-pointer import entirely + +Also: the failing op named in an error is usually just whichever op was in +flight when the context died, not the op at fault. Chasing a specific kernel or +shape is a dead end. + +### Useful corelib switches when debugging + +| Variable | Effect | +|---|---| +| `RYZENAI_CORELIB_TRACE_XRT` | trace XRT calls | +| `RYZENAI_CORELIB_SAFE_MODE` | force staging buffers, no host-pointer import | +| `RYZENAI_CORELIB_NO_RUN_POOL` | disable run-object reuse | +| `RYZENAI_CORELIB_NO_CRASH_HANDLER` | leave crashes to the debugger | + +### Other rough edges + +- `home_install.sh`'s `needs_configure` does not check that the cached + `CMAKE_HOME_DIRECTORY` matches `$SRC_DIR`, so a build tree configured from a + different source dir is reused rather than reconfigured. +- `src/src/main.cpp` is **CRLF**. Editing it with a tool that writes LF rewrites + every line and produces a ~1800-line diff. Check `git diff --stat -w` if a + diff looks impossibly large. diff --git a/src/home_install.sh b/src/home_install.sh index daeacf64f..fca6889e9 100755 --- a/src/home_install.sh +++ b/src/home_install.sh @@ -267,14 +267,14 @@ elif [[ -f "$CACHE_FILE" ]] && grep -q '^FLM_ENABLE_RAI:BOOL=ON' "$CACHE_FILE"; # returns -- no warning, no error. # # It only exists when DD was configured with DD_MDS_IN_BINS_DLL=ON, which - # moves every transaction/mds/*.elf -- i.e. the whole AIE4 kernel set -- out + # moves every transaction/mds/*.elf -- i.e. the whole AIE_NEXT kernel set -- out # of the static transaction lib and into this library. Skipping it leaves - # the AIE4 shape table empty, and the first matmul dies with + # the AIE_NEXT shape table empty, and the first matmul dies with # Target Shape (K: 3072, N: 3072, Gs: 64) ... not supported in this # supported shape list. Shape list size: 0 # which reads like an unsupported model rather than a missing file. if [[ -n "$DD_CORE_DIR" && -f "$DD_CORE_DIR/libdyn_bins.so" ]]; then - echo "[home_install] + libdyn_bins.so (dlopen'd AIE4 kernel package)" + echo "[home_install] + libdyn_bins.so (dlopen'd AIE_NEXT kernel package)" install -m 0755 "$DD_CORE_DIR/libdyn_bins.so" "$RAI_LIB_DEST/" fi else @@ -319,7 +319,7 @@ XRT_ROOT="$XRT_ROOT" # that root only because the plugin directory exists -- and put the same # directory on LD_LIBRARY_PATH so the linked-in libxrt_coreutil comes from # that install too. Without the pin nothing reports a missing root: the NPU -# just comes up with no driver, corelib says "no AIE4 hw_context +# just comes up with no driver, corelib says "no AIE_NEXT hw_context # (unordered_map::at)" and xrt::device(0) says "No such library # .../libxrt_core.so.2". if [[ -f "\$XRT_DIR/setup.sh" ]]; then From ecebd2ee26601d76e0814236e8638441846acf0c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Thu, 24 Sep 2026 17:21:13 -0700 Subject: [PATCH 12/17] test(rai): follow the rotary tables onto host views cb8a19dd moved flat_mha's cosine/sine tables from device tensors to host views and added create_host_view to the resolved symbol table, but the suite still described the old shape: test_corelib_api counted 26 symbols, and test_phi4_engine counted 74 device tensors and located writes and the K/V caches by their old positions. Both failed on the PR head. The fake corelib now implements create_host_view and, like the real one, rejects anything but a host view in flat_mha's rotary slots, so a regression back to device tensors fails the engine test instead of passing silently. The cache tests find layer 0's K cache by shape rather than by create index, and validation is checked to create no host views. Carried from medusa's 4ed7c0b3, test side only; the engine change itself is cb8a19dd's. Co-authored-by: Cursor --- src/test/phi4_rai/fake_corelib.cpp | 28 +++++++++++++++++++++++ src/test/phi4_rai/fake_corelib.hpp | 1 + src/test/phi4_rai/test_corelib_api.cpp | 14 +++++++----- src/test/phi4_rai/test_phi4_engine.cpp | 31 ++++++++++++++++++++++---- src/test/phi4_rai/test_phi4_gguf.cpp | 1 + 5 files changed, 65 insertions(+), 10 deletions(-) diff --git a/src/test/phi4_rai/fake_corelib.cpp b/src/test/phi4_rai/fake_corelib.cpp index 251da0118..39cf0c672 100644 --- a/src/test/phi4_rai/fake_corelib.cpp +++ b/src/test/phi4_rai/fake_corelib.cpp @@ -220,6 +220,23 @@ struct TypedFake { state.tensor_windows.push_back({parent, object->shape, offset, object}); } return status; + } else if constexpr (std::is_same_v) { + const auto status = Status(Tag::name); + const auto type = std::get<0>(arguments); + const auto* shape = std::get<1>(arguments); + const auto shape_len = std::get<2>(arguments); + const void* data = std::get<3>(arguments); + auto* out = std::get<4>(arguments); + if (out) *out = nullptr; + if (status == ryzenai_corelib_status_success && out && shape && data) { + auto* object = static_cast(NewObject("host_view")); + object->data_type = type; + object->shape.assign(shape, shape + shape_len); + object->byte_size = Elements(object->shape) * TypeBytes(type); + *out = object; + state.host_view_creates.push_back({type, object->shape, object}); + } + return status; } else if constexpr (std::is_same_v) { const auto status = Status(Tag::name); if (status == ryzenai_corelib_status_success && std::get<0>(arguments) && std::get<1>(arguments)) @@ -434,6 +451,16 @@ struct TypedFake { } const auto status = Status(Tag::name); if (status != ryzenai_corelib_status_success) return status; + if constexpr (std::is_same_v) { + // Real corelib reads the rotary tables on the host and rejects + // anything but a host view for them. + for (void* table : {std::get<5>(arguments), std::get<6>(arguments)}) { + if (!table || static_cast(table)->kind != "host_view") { + state.detail = "handle is not a class RyzenAI::CoreLib::HostView"; + return ryzenai_corelib_status_bad_argument; + } + } + } fake_corelib::DispatchRecord record{}; record.thread_id = std::this_thread::get_id(); record.kind = std::is_same_v ? "matmul" : @@ -543,6 +570,7 @@ void Reset() { state.matmul_n_delta = 0; state.pad_row_overrides.clear(); state.tensor_creates.clear(); + state.host_view_creates.clear(); state.tensor_windows.clear(); state.weight_creates.clear(); state.weight_from_file.clear(); diff --git a/src/test/phi4_rai/fake_corelib.hpp b/src/test/phi4_rai/fake_corelib.hpp index c5ee85935..556078df1 100644 --- a/src/test/phi4_rai/fake_corelib.hpp +++ b/src/test/phi4_rai/fake_corelib.hpp @@ -126,6 +126,7 @@ struct State { std::unordered_map> pad_row_overrides; std::vector tensor_creates; + std::vector host_view_creates; std::vector tensor_windows; std::vector weight_creates; std::vector weight_from_file; diff --git a/src/test/phi4_rai/test_corelib_api.cpp b/src/test/phi4_rai/test_corelib_api.cpp index a81a68149..5b91d05c5 100644 --- a/src/test/phi4_rai/test_corelib_api.cpp +++ b/src/test/phi4_rai/test_corelib_api.cpp @@ -17,6 +17,7 @@ namespace { using flm::corelib::CorelibApi; using flm::corelib::CorelibError; using flm::corelib::CorelibRuntime; +using flm::corelib::UniqueHostView; using flm::corelib::UniqueMatMulWeights; using flm::corelib::UniqueSsMlpWeights; using flm::corelib::UniqueStream; @@ -40,7 +41,7 @@ void TestVersionIsResolvedBeforeEveryOtherSymbol() { fake_corelib::Reset(); ValidApi(); const auto& order = fake_corelib::GetState().resolution_order; - TEST_REQUIRE(order.size() == 26); + TEST_REQUIRE(order.size() == 27); TEST_REQUIRE(order.front() == "ryzenai_corelib_get_version"); } @@ -71,7 +72,7 @@ void TestMajorMinorAndPatchMismatchesAreRejectedWithBothVersions() { void TestEveryRequiredSymbolIsResolvedExactlyOnce() { fake_corelib::Reset(); ValidApi(); - TEST_REQUIRE(fake_corelib::GetState().resolution_counts.size() == 26); + TEST_REQUIRE(fake_corelib::GetState().resolution_counts.size() == 27); for (const auto& [name, count] : fake_corelib::GetState().resolution_counts) { (void)name; TEST_REQUIRE(count == 1); @@ -85,11 +86,11 @@ void TestEveryResolvedFakeFunctionUsesItsExactAbi() { fake_corelib::GetState().default_status = ryzenai_corelib_status_bad_argument; fake_corelib::GetState().selftest_status = ryzenai_corelib_status_bad_argument; const auto statuses = fake_corelib::CallEveryResolvedFunction(api->functions()); - TEST_REQUIRE(statuses.size() == 20); + TEST_REQUIRE(statuses.size() == 21); TEST_REQUIRE(std::all_of(statuses.begin(), statuses.end(), [](auto status) { return status == ryzenai_corelib_status_bad_argument; })); - TEST_REQUIRE(fake_corelib::GetState().call_counts.size() == 26); + TEST_REQUIRE(fake_corelib::GetState().call_counts.size() == 27); for (const auto& [name, count] : fake_corelib::GetState().call_counts) { (void)name; TEST_REQUIRE(count == 1); @@ -188,13 +189,14 @@ void TestEveryUniqueObjectReleasesExactlyOnceAfterMoves() { assigned = std::move(moved); UniqueStream stream(api, fake_corelib::MakeObject()); UniqueTensorWindow window(api, fake_corelib::MakeObject()); + UniqueHostView host_view(api, fake_corelib::MakeObject()); UniqueMatMulWeights matmul(api, fake_corelib::MakeObject()); UniqueSsMlpWeights ssmlp(api, fake_corelib::MakeObject()); TEST_REQUIRE(!first && !moved && assigned); - TEST_REQUIRE(api->live_object_count() == 5); + TEST_REQUIRE(api->live_object_count() == 6); TEST_REQUIRE(fake_corelib::GetState().releases == 0); } - TEST_REQUIRE(fake_corelib::GetState().releases == 5); + TEST_REQUIRE(fake_corelib::GetState().releases == 6); TEST_REQUIRE(api->live_object_count() == 0); } diff --git a/src/test/phi4_rai/test_phi4_engine.cpp b/src/test/phi4_rai/test_phi4_engine.cpp index d87e035c0..8b9cc4c88 100644 --- a/src/test/phi4_rai/test_phi4_engine.cpp +++ b/src/test/phi4_rai/test_phi4_engine.cpp @@ -77,7 +77,7 @@ void TestEngineCreatesOneStreamAndPersistentHelperSizedTensors() { Harness h; const auto& state = fake_corelib::GetState(); TEST_REQUIRE(state.call_counts.at("ryzenai_corelib_create_stream") == 1); - TEST_REQUIRE(state.tensor_creates.size() == 74); + TEST_REQUIRE(state.tensor_creates.size() == 72); TEST_REQUIRE(state.tensor_creates[0].shape == std::vector({4096, 3072})); TEST_REQUIRE(state.tensor_creates[3].shape == std::vector({4096, 3072})); TEST_REQUIRE(state.tensor_creates[4].shape == std::vector({4096, 1024})); @@ -85,6 +85,18 @@ void TestEngineCreatesOneStreamAndPersistentHelperSizedTensors() { TEST_REQUIRE(state.tensor_creates[7].shape == std::vector({1, 200064})); } +void TestRopeTablesAreHostViewsNotDeviceTensors() { + Harness h; + const auto& views = fake_corelib::GetState().host_view_creates; + TEST_REQUIRE(views.size() == 2); + for (const auto& view : views) { + TEST_REQUIRE(view.data_type == ryzenai_corelib_data_type_fp32); + TEST_REQUIRE(view.shape == std::vector({4096, 48})); + } + (void)h.engine->forward(1); + TEST_REQUIRE(!h.engine->poisoned()); +} + void TestEngineAllocatesMaximaAcrossAllRowsAndConsumers() { Harness h([](auto& state) { state.pad_row_overrides["matmul-3072"][2048] = 5000; @@ -389,7 +401,7 @@ void TestPrefillDecodesEmbeddingRowsAndAdvancesPosition() { const auto logits = h.engine->prefill(ids); TEST_REQUIRE(logits.size() == 200064); TEST_REQUIRE(h.engine->get_current_context_length() == 3); - TEST_REQUIRE(fake_corelib::GetState().tensor_writes[2].source_type == ryzenai_corelib_data_type_fp32); + TEST_REQUIRE(fake_corelib::GetState().tensor_writes[0].source_type == ryzenai_corelib_data_type_fp32); } void TestDecodeUsesOneRowAndAdvancesPosition() { @@ -608,9 +620,19 @@ void WriteCacheRow(Harness& h, std::size_t tensor_index, int position, } } +/// \brief index of layer 0's K cache; its V cache is the next create +std::size_t FirstCacheCreate() { + const auto& creates = fake_corelib::GetState().tensor_creates; + const auto found = std::find_if(creates.begin(), creates.end(), [](const auto& record) { + return record.shape == std::vector({8, 4096, 128}); + }); + TEST_REQUIRE(found != creates.end()); + return static_cast(found - creates.begin()); +} + void TestGetKCacheGathersHeadMajorPosition() { Harness h; - WriteCacheRow(h, 10, 7, 100); + WriteCacheRow(h, FirstCacheCreate(), 7, 100); const auto result = h.engine->get_k_cache(0, 7); const auto* bits = reinterpret_cast(result.data()); for (std::size_t head = 0; head < 8; ++head) @@ -620,7 +642,7 @@ void TestGetKCacheGathersHeadMajorPosition() { void TestGetVCacheGathersHeadMajorPosition() { Harness h; - WriteCacheRow(h, 11, 9, 200); + WriteCacheRow(h, FirstCacheCreate() + 1, 9, 200); const auto result = h.engine->get_v_cache(0, 9); const auto* bits = reinterpret_cast(result.data()); for (std::size_t head = 0; head < 8; ++head) @@ -704,6 +726,7 @@ void TestTenSequentialLoadsReleaseEveryObjectAndNeverEmitAllZeroLogits() { int main() { #define RUN_TEST(name) RunTest(&name, #name) RUN_TEST(TestEngineCreatesOneStreamAndPersistentHelperSizedTensors); + RUN_TEST(TestRopeTablesAreHostViewsNotDeviceTensors); RUN_TEST(TestEngineAllocatesMaximaAcrossAllRowsAndConsumers); RUN_TEST(TestEngineCreatesExactly129MatmulAnd32SsmlpWeights); RUN_TEST(TestEveryProjectionUsesQ8RequantizedGroup64WithThreadHint); diff --git a/src/test/phi4_rai/test_phi4_gguf.cpp b/src/test/phi4_rai/test_phi4_gguf.cpp index 42aebdbc0..018e35a0e 100644 --- a/src/test/phi4_rai/test_phi4_gguf.cpp +++ b/src/test/phi4_rai/test_phi4_gguf.cpp @@ -535,6 +535,7 @@ void TestValidationCreatesNoCorelibObjects() { for (const auto name : {"ryzenai_corelib_create_stream", "ryzenai_corelib_create_device_tensor", "ryzenai_corelib_create_tensor_window", + "ryzenai_corelib_create_host_view", "ryzenai_corelib_matmul_bf16_weights_create_gguf_requantized", "ryzenai_corelib_ssmlp_bf16_weights_create_gguf_requantized"}) TEST_REQUIRE(fake_corelib::GetState().call_counts[name] == 0); From 11ac533d13f790cd35c95c0e55eb9722b983911a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Thu, 24 Sep 2026 17:22:02 -0700 Subject: [PATCH 13/17] test(phi4): expect the backend error 40aa9fe4 now raises 40aa9fe4 reworded the unregistered-backend error to "is not available for model family ''. It provides: ..." and moved test_model_backend to match, but test_phi4_frontend still looked for the old "not compiled into this build", so both its ON and OFF builds failed. The OFF case also checks the provider list is exactly flm, which is the point of that test: a build without corelib registers no rai engine. Co-authored-by: Cursor --- src/test/phi4_rai/test_phi4_frontend.cpp | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/src/test/phi4_rai/test_phi4_frontend.cpp b/src/test/phi4_rai/test_phi4_frontend.cpp index 88edfaf92..28ff05b12 100644 --- a/src/test/phi4_rai/test_phi4_frontend.cpp +++ b/src/test/phi4_rai/test_phi4_frontend.cpp @@ -480,7 +480,7 @@ void TestUnknownBackendIsAnError() { // engine is constructed. const auto message = RequireThrows( [&] { (void)Load(package, ModelInfo(), -1, false, nullptr, "other"); }); - RequireContains(message, "not compiled into this build"); + RequireContains(message, "is not available for model family 'phi4'"); RequireContains(message, "other"); TEST_REQUIRE(g_factory.legacy_calls == 0 && g_factory.rai_calls == 0); } @@ -489,9 +489,11 @@ void TestFeatureOffRejectsRaiTagWithoutIncludingCorelibHeaders() { #if !defined(FLM_ENABLE_RAI) TempPackage package; FactoryScope scope; - RequireContains(RequireThrows([&] { + const auto message = RequireThrows([&] { (void)Load(package, ModelInfo(), -1, false, nullptr, kRai); - }), "not compiled into this build"); + }); + RequireContains(message, "is not available for model family 'phi4'"); + RequireContains(message, "It provides: flm"); TEST_REQUIRE(g_factory.legacy_calls == 0 && g_factory.rai_calls == 0); #endif } From 41a057a9e784600597963db72a203794926f1b7f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Thu, 24 Sep 2026 17:22:53 -0700 Subject: [PATCH 14/17] fix(automodel): keep prefill logits owned until they are sampled _chunked_insert took the last chunk's logits with `y = chunk_y`. buffer's copy assignment is shallow and does not take ownership, so once chunk_y went out of scope at the end of the loop body, y pointed at freed memory and the first token of a reply was sampled from it. The flm engines never showed it: they return views of memory they keep. The rai phi4 engine returns logits it owns, so on rai the first token came from whatever the heap had put there -- a first reply of "!4" to "What is 2+2?", and fresh processes answering with "!", "%", "…" or "![](...". It now moves, which for a non-owning buffer is the same as the copy was. The phi4 frontend test's fake engine already returns owning logits; its sampler stub now counts samples taken from a buffer that no longer owns them, and a new test drives the chunked path and requires none. (cherry picked from commit 23b4fa3296c0812dbddad3051f9922466e142e03) Co-authored-by: Cursor --- src/common/AutoModel/automodel.cpp | 5 ++++- src/test/phi4_rai/test_phi4_frontend.cpp | 26 +++++++++++++++++++++++- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/src/common/AutoModel/automodel.cpp b/src/common/AutoModel/automodel.cpp index 81559a9e1..f5281ebd9 100644 --- a/src/common/AutoModel/automodel.cpp +++ b/src/common/AutoModel/automodel.cpp @@ -448,7 +448,10 @@ buffer AutoModel::_chunked_insert(chat_meta_info_t& meta_info, std::vector } buffer chunk_y = this->lm_engine->prefill(chunk_tokens, (i == 0)? payload : nullptr); if (i == chunks - 1) { - y = chunk_y; + // Moved, not copied: buffer's copy is shallow and does not take + // ownership, so a copy of logits the engine owns (the rai + // engines return them that way) dangles once chunk_y goes. + y = std::move(chunk_y); } } } diff --git a/src/test/phi4_rai/test_phi4_frontend.cpp b/src/test/phi4_rai/test_phi4_frontend.cpp index 28ff05b12..83d0e1ef5 100644 --- a/src/test/phi4_rai/test_phi4_frontend.cpp +++ b/src/test/phi4_rai/test_phi4_frontend.cpp @@ -44,6 +44,7 @@ std::vector g_encoded_tokens; std::vector g_samples; std::vector g_opened_paths; std::size_t g_sample_index{}; +int g_unowned_samples{}; class FakeEngine final : public causal_lm { public: @@ -287,7 +288,11 @@ Sampler::Sampler(int features, sampler_config& config) logits.resize(1); counters.resize(1); token_positions.resize(1, -1); } void Sampler::reset_penalties() {} -int Sampler::sample(buffer&) { +int Sampler::sample(buffer& logits) { + // FakeEngine hands back logits it allocated, as the rai engines do. buffer's + // copy is shallow and non-owning, so a buffer that no longer owns them by the + // time they are sampled is reading memory its owner has already freed. + if (!logits.is_owner()) ++g_unowned_samples; if (g_sample_index < g_samples.size()) return g_samples[g_sample_index++]; return 7; } @@ -389,6 +394,24 @@ void TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib() { TEST_REQUIRE(g_factory.rai_calls == 0); } +void TestPrefillLogitsAreStillOwnedWhenSampled() { + TempPackage package; + FactoryScope scope; + auto model = Load(package, ModelInfo()); + g_encoded_tokens = {1, 2, 3}; + g_samples = {7}; + g_sample_index = 0; + g_unowned_samples = 0; + auto meta = Meta(); + // A prefill limit of 512 or more takes the chunked path, where the logits + // are handed out of the last chunk. + meta.max_prefill_len = 4096; + auto input = Input(1); + TEST_REQUIRE(model->insert(meta, input)); + TEST_REQUIRE(g_factory.engine->prefill_calls == 1); + TEST_REQUIRE(g_unowned_samples == 0); +} + void TestEnabledBuildStartsAndRunsLegacyPhi4WhenCorelibDllIsMissing() { TempPackage package; FactoryScope scope; @@ -864,6 +887,7 @@ int main() { RunTest(TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable, "TestCancellationAndCapacityErrorsLeaveTheServerQueueUsable"); #else RunTest(TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib, "TestDefaultBuildCanConstructAndRunLegacyPhi4WithoutCorelib"); + RunTest(TestPrefillLogitsAreStillOwnedWhenSampled, "TestPrefillLogitsAreStillOwnedWhenSampled"); RunTest(TestFeatureOffRejectsRaiTagWithoutIncludingCorelibHeaders, "TestFeatureOffRejectsRaiTagWithoutIncludingCorelibHeaders"); #endif std::cout << "test_phi4_frontend: PASS\n"; From 19f1f190205966ac290a759318a6f68f86c0fc4a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Thu, 24 Sep 2026 17:23:43 -0700 Subject: [PATCH 15/17] fix(rai): drop the bring-up DEBUG prints 2bf92e8f meant to remove 2bf92e8f says it drops three DEBUG prints left over from bringing the backend up, but its diff removes none: every rai build still printed "RAI enabled!" and "got device from corelib" on every command, `flm list` included. The third, "corelib reports no NPU device", only repeated what npu_open_error already carries to the "rai backend unavailable" line and the no-device failure below it. Co-authored-by: Cursor --- src/src/main.cpp | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/src/src/main.cpp b/src/src/main.cpp index e15aceba1..f4218b06e 100644 --- a/src/src/main.cpp +++ b/src/src/main.cpp @@ -610,9 +610,6 @@ int main(int argc, char* argv[]) { // backend is gone, the rest of the build is not. std::string corelib_note; - - header_print("DEBUG", "RAI enabled!"); - try { const auto runtime = flm::corelib::CorelibRuntime::GetOrCreate(std::filesystem::path(exe_dir)); @@ -622,14 +619,9 @@ int main(int argc, char* argv[]) { // never completes. GetOrCreate holds the runtime process-wide, so the // device stays valid until RaiProcessGuard tears it down at exit. npu_device = flm::corelib::SharedDevice(*runtime); - if (npu_device == nullptr) { + if (npu_device == nullptr) npu_open_error = "corelib started but reports no NPU device on this machine"; - header_print("DEBUG", "corelib reports no NPU device on this machine"); - } - else { - header_print("DEBUG", "got device from corelib"); - } } catch (const std::exception& e) { // A box with no NPU must still run `flm list`/`pull`/`version`, so this // stays a null device rather than an error, as in the non-rai path. From f217211175c388f175d7ba701e2e3320ea41091d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E2=80=9CChi?= Date: Thu, 24 Sep 2026 23:13:01 -0700 Subject: [PATCH 16/17] test(phi4): make the every-tensor-role GGUF rejection test opt-in TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole writes four full-size Phi-4 fixtures for every required tensor role and keeps up to five of them alive at once. On a shared build box that is enough to run the volume out of space, and the resulting truncated fixtures fail as "out-of-file range" rather than as a disk error. It now runs only with FLM_PHI4_GGUF_EVERY_ROLE set. The rejection paths it sweeps are still covered one role at a time by the mixed-quantization, tied-embedding and rope-factor tests that stay in the default run. Co-authored-by: Cursor --- src/test/phi4_rai/test_phi4_gguf.cpp | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/test/phi4_rai/test_phi4_gguf.cpp b/src/test/phi4_rai/test_phi4_gguf.cpp index 018e35a0e..bf1e6b7b4 100644 --- a/src/test/phi4_rai/test_phi4_gguf.cpp +++ b/src/test/phi4_rai/test_phi4_gguf.cpp @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -563,7 +564,10 @@ int main() { RUN(TestViewsPointIntoTheReadOnlyMapping); RUN(TestAcceptsExactPhi3Phi4Contract); RUN(TestRejectsWrongArchitectureAndEveryDimension); - RUN(TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole); + // Opt-in: it writes four full-size (~4 GB) fixtures for every tensor role, + // with up to five alive at once, which a shared build box cannot absorb. + if (std::getenv("FLM_PHI4_GGUF_EVERY_ROLE")) + RUN(TestRejectsMissingWrongTypeWrongShapeAndWrongLengthForEveryTensorRole); RUN(TestRejectsMixedQuantizationAndOutputWeightPresence); RUN(TestRequiresTiedQ8TokenEmbeddingAsLmHead); RUN(TestRequiresOriginal4096WindowAndValidatesLongRopeFactors); From 7ad2cdc717768baa5bc26603198dcbb7ba6df3ba Mon Sep 17 00:00:00 2001 From: alfxu_amdeng Date: Fri, 25 Sep 2026 08:26:04 -0700 Subject: [PATCH 17/17] feat: ensure main branch compatibility --- src/include/utils/npu_platform.hpp | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/include/utils/npu_platform.hpp b/src/include/utils/npu_platform.hpp index 5b1730a02..c49100557 100644 --- a/src/include/utils/npu_platform.hpp +++ b/src/include/utils/npu_platform.hpp @@ -43,7 +43,13 @@ constexpr std::string_view platform_id(npu_platform platform) { /// offers nothing here. That is not a broken install, it is an install /// for silicon none of its models were built for, and model_list says so /// in one line. Set this back to aie2p to get those 42 models back. -constexpr npu_platform default_npu_platform() { return npu_platform::aie_next; } +constexpr npu_platform default_npu_platform() { +#if defined(FLM_ENABLE_RAI) + return npu_platform::aie_next; +#else + return npu_platform::aie2p; +#endif +} /// \brief ask the machine which NPU generation it has /// \return the generation of the NPU in this host