feat(ENG-HOST-EMBEDDING): gather the token table on the host

[iiLaurens]

Row

ENG-HOST-EMBEDDING (new row in this PR) — one row per PR. Spec:
.agents/specs/host-embedding.md.

Before starting

• Issue/PR search: no issue existed for a host-resident embedding gather; vLLM
  has no equivalent, so this is a vllm.cpp-original capability whose shape comes
  from llama.cpp's ggml_get_rows (the secondary oracle next to the pinned
  IQ4_NL/Q5_0 vectors). Filed ISSUE-LOCAL-01M3QETSTM8X8AJKBM9QCTGBKM.
• Pull request shape selected at row claim: one PR for the spec and the
  implementation.
• New row because none of the existing rows owns it: ENG-WEIGHT-OFFLOAD is
  vLLM's cpu_offload_gb mirror and ENG-EXPERT-STREAM-DEVICE is the
  host-addressable destination half; this is a host-side gather, not either.
  The category counts and the total in .agents/engine-matrix.md move with it.
• Anchors inspected: the pre-change gather (ResidentWeight +
  vt::Embedding), docs/ENVIRONMENT.md's knob table, and the pinned goldens
  in tests/vt/iq4nl_q5_0_golden_vectors.h.

What changed

VT_HOST_EMBEDDING=1 keeps the token table in host RAM and gathers the requested
rows on a cached CPU queue, then copies the [T,H] result to the device. The CPU
vt::Embedding kernel decodes one row per id, so every table residency the
loaders produce works (bf16/f16/f32 and the GGUF block formats); a bf16 table
takes a byte-copy fast path. A new EmbedGather seam owns both arms, so every
dense forward that owns a device table gets it and the device arm is unchanged
when the flag is off or the host bytes are gone. The async runner's
device-resident id override is consumed before the gather.

Evidence

New and first coverage for the arm, tests/vllm/models/test_host_embedding.cpp:

$ ./build/tests/test_host_embedding
[host-embed] CPU row gather (T=5 dtype=iq4_nl out=bf16)
test cases:   5 |   5 passed | 0 failed | 0 skipped
assertions: 852 | 852 passed | 0 failed

The gate has two halves. The rows are compared against the SAME pinned IQ4_NL
golden bits the device-side test_ops_embedding_quant gate uses (bf16, rounded
once), and the case asserts the arm actually ran — the CPU device arm aliases the
host bytes instead of staging, so d_dev == nullptr alone cannot tell the arms
apart on this backend; the [host-embed] banner is captured and checked, and the
other cases check HostEmbedInto's return value.

MUTATION, the guarantee this pins — disable the arm and rebuild (the flag is
process-static):

$ sed -i 's/VT_HOST_EMBEDDING", "1"/VT_HOST_EMBEDDING", "0"/' tests/vllm/models/test_host_embedding.cpp && rebuild
[doctest] test cases:   5 |   2 passed | 3 failed | 0 skipped
[doctest] assertions: 852 | 848 passed | 4 failed

The device arm is untouched:

$ ./build/tests/test_ops_embedding_quant
test cases: 6 | 6 passed | 0 failed
assertions: 1637 | 1637 passed | 0 failed

• python3 scripts/check-agent-record.py — OK (ENGINE=180 MODEL=384 QUANT=87 KERNEL=60 BACKEND=90); the row, its structured spec, its claim
  record, the matrix counts and the pinned ENGINE_ROWS land together
• the ENGINE_ROWS bump carries its executable evidence:
  tests/scripts/test_agent_record.py::HostEmbeddingRowIsCounted (3 tests,
  green) asserts the row exists once, names specs/host-embedding.md and
  CLAIM-ENG-HOST-EMBEDDING, and that the spec carries all nine structured
  sections. Under the mutation that drops the claim owner, the class reports
  FAILED (failures=1, errors=1); restored, 3/3 green
  (check-pr-size.py now exits 0 for this branch)
• tests that cover this change: tests/vllm/models/test_host_embedding.cpp,
  tests/vt/test_ops_embedding_quant.cpp
• docs/ENVIRONMENT.md documents both new variables in this change
  (check-env-doc reports only the pre-existing VT_VK_* gap)
• full scripts/agent-preflight.sh — RED ON THIS BOX AT BASE, attributed:
  every failing gate was re-run on a clean origin/main worktree and fails
  there too (check-release-binary-contract, check-windows-release-state,
  check-benchmark-index, check-env-doc, check-test-registration,
  check-oracle-pins — the latter on the e126687a9a vs a7c23ac96d pin
  mismatch that predates this branch — and check-symbol-anchors). The full
  ctest failures are attributed in PR fix(LOAD-MODELOPT-NVFP4-BORROW): one owner per device resident #3355's body; the 12 of them reproduce
  on base binaries built from b45a94273.

Review follow-up, verified on Linux and on the GPU

The review's two findings were fixed in d686c8582 and re-verified here:

• The oversized override can no longer write past the [T] id buffer: the host
  arm splices through the same detail::ApplyDeviceTokenIds the device arm uses.
• The Windows compile blocker (::setenv in an unconditional target) is gone
  via vllm_test::SetEnv.

Linux, Release CPU build:

$ ./build/tests/test_host_embedding
test cases:   7 |   7 passed | 0 failed | 0 skipped
assertions: 866 | 866 passed | 0 failed

The two new cases are the ones the review asked for: a one-id override over a
three-row host upload, which must yield [1, 1, 2] (the shorter-prefix case
preserving the host tail), and a two-id override over a one-row upload, which
must throw.

Linux + the local sm_120a card, on the official checkpoint, a scratch tree
carrying this fix plus #3355/#3360 (all three produce identical tokens):

$ VT_HOST_EMBEDDING=0 ... --max-tokens 8   -> device_upload=12.044 GiB  "The capital of Germany is"
$ VT_HOST_EMBEDDING=1 VT_HOST_EMBED_TRACE=1 ...
   [host-embed] T=5 override=1
   [host-embed] bf16 row copy (T=5)
                                              -> device_upload= 9.676 GiB  "The capital of Germany is"

override=1 is the async runner's spliced device ids — the exact path the
review found unbounded. It runs in production and the tokens are identical to
the flag-off arm.

Speed claims

• This PR makes NO speed claim. It is an opt-in memory arm; the device-memory
  saving and any decode effect are owed and unmeasured.

Honest gaps

• Measured on the official ukisai/Swift-1.5-Qwen3.8-27b-NVFP4 safetensors
  checkpoint (21.94 GB) on the local sm_120a 24 GiB card, same binary, only
  VT_HOST_EMBEDDING differs (a scratch tree carrying this change plus fix(LOAD-MODELOPT-NVFP4-BORROW): one owner per device resident #3355's
  NVFP4 ownership fix, which this artifact needs to fit — see PR fix(LOAD-MODELOPT-NVFP4-BORROW): one owner per device resident #3355):

  $ VT_HOST_EMBEDDING=1 VT_HOST_EMBED_TRACE=1 ... --max-tokens 8
  [host-embed] T=5 override=1
  [host-embed] bf16 row copy (T=5)
  [vt load] bytes@exit  host_copy=7.621 GiB borrowed=12.004 GiB device_upload=9.676 GiB
  "The capital of Germany is"   1.913 tok/s
  
  $ VT_HOST_EMBEDDING=0 ... --max-tokens 8
  [vt load] bytes@exit  host_copy=7.621 GiB borrowed=12.004 GiB device_upload=12.044 GiB
  "The capital of Germany is"   1.826 tok/s
  

  2.368 GiB less device_upload with identical tokens, and the override=1
  banner shows the async runner's spliced device ids were consumed before the
  gather. Not a throughput claim (an 8-token decode is not a speed gate).

• The arm is opt-in (flag off by default). Nothing in the shipped default path
  changes.

• The mutation evidence is author-run; a fresh reviewer over the immutable head
  still owes its own pass.

• scripts/agent-preflight.sh on this box is red at base (35 gates): record
  drift, check-test-registration's CMake probe, the oracle pin mismatch
  (e126687a9a vs a7c23ac96d), and the missing file tool. Each failing gate
  was re-run on a clean origin/main worktree and fails there too; none of them
  touches a file this PR changes.

Refs ISSUE-LOCAL-01M3QETSTM8X8AJKBM9QCTGBKM

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:opencode-go/deepseek-v4.1-flash [pi]

[localai-org-maint-bot]

The new host-embedding path bypasses an existing safety check. In HostEmbedInto, dids holds T int32 identifiers, but the raw copy uses override_ids.count without checking that it fits. With T=1 and a two-element DeviceTokenIdsScope, this requests an 8-byte write into a 4-byte buffer. The device arm's detail::ApplyDeviceTokenIds already rejects that mismatch, and provides an overload accepting the taken override. Please route the host arm through that helper before downloading the identifiers, and add an oversized-override regression plus a shorter-prefix case preserving the padded host tail. The new suite currently never supplies an override.

There is also a Windows compile blocker: EnableHostEmbedding calls ::setenv twice outside any platform guard, while test_host_embedding is an unconditional target. Use the existing portable vllm_test::SetEnv helper from support/test_env.h.

These are focused static findings at 3cd5700, based on the new helper/tests and the existing override contract and implementation. I have not completed acceptance of all model call sites or executed the C++/device gates; this host lacks the required toolchain. @mudler @richiejp: these paths need correction before merge.

[iiLaurens]

Repairs pushed at d686c8582.

Override bounds. HostEmbedInto now splices the taken override through
detail::ApplyDeviceTokenIds — the same body the device arm uses — instead of
its own unchecked Copy. An override longer than the embed input throws with
the caller's name; a shorter one replaces its prefix and the padded host tail
stays. Two new cases in test_host_embedding pin both boundaries, and the
target now takes ${CMAKE_SOURCE_DIR}/src on its include path for the internal
detail::DeviceTokenIdsScope declaration.

Windows compile. EnableHostEmbedding uses vllm_test::SetEnv from
tests/support/test_env.h (the _putenv_s arm) instead of POSIX ::setenv.

Evidence: Visual Studio 2022 Build Tools 17.14, x64 Release, CPU-only —
test_host_embedding builds and runs 7/7 cases, 862/862 assertions;
test_ops_embedding_quant stays at 1637/1637.

[iiLaurens]

Addressed in d686c8582.

• The unbounded copy is gone. HostEmbedInto now splices the override through the same detail::ApplyDeviceTokenIds the device arm uses, so a count above T throws with the caller's name instead of writing past the [T] buffer, and a shorter override replaces its prefix while the host upload's tail is preserved.
• The Windows compile blocker is gone. The initializer uses vllm_test::SetEnv from support/test_env.h, and the target takes ${CMAKE_SOURCE_DIR}/src so the detail::DeviceTokenIdsScope seam resolves (same reach as the other detail:: gates).

Both cases you asked for are covered:

• host embedding: a device override splices its rows over the host upload — one id over a three-row host upload, output must read [1, 1, 2] (the padded tail case).
• host embedding: an override longer than the embed input is refused — two ids over one row, must throw.

Verified on Linux (Release CPU build):

$ ./build/tests/test_host_embedding
test cases:   7 |   7 passed | 0 failed | 0 skipped
assertions: 866 | 866 passed | 0 failed

and end to end on the local sm_120a card with the official ukisai/Swift-1.5-Qwen3.8-27b-NVFP4 safetensors checkpoint (scratch tree carrying #3355 and #3360 so the 27B fits a 24 GiB card; same binary, only the flag differs):

$ VT_HOST_EMBEDDING=0 ... --max-tokens 8
[vt load] bytes@exit  device_upload=12.044 GiB
"The capital of Germany is"

$ VT_HOST_EMBEDDING=1 VT_HOST_EMBED_TRACE=1 ... --max-tokens 8
[host-embed] T=5 override=1
[host-embed] bf16 row copy (T=5)
[vt load] bytes@exit  device_upload=9.676 GiB
"The capital of Germany is"

override=1 is the async runner's spliced device ids — the exact path the review found unbounded. It runs in production with the bound in place, and the tokens are identical to the flag-off arm.