[GoldenLeafBird]

Problem and result

Fixes #3431 and #2960, which report the same fault. On Windows with the HIP engine, about one engine start in ten answers every long prompt wrong until the engine restarts. The answers are fluent text about something that is not in the prompt, with HTTP 200 and no error in the log. The only trace is KV streaming: 100.00% of 0 block reads for each request. This line shows on short prompts too, also when their answer is correct.

Cause (measured on an RX 7900 XTX with engine 0.1.54, v0.2.8):

• PinnedArena registers the expert arena with cudaHostRegister(Portable | Mapped), here in 36 slices. On Windows the HIP runtime gives each registered range a second address for the GPU (cudaHostGetDevicePointer). That address differs from the host address and lies low in the address space (seen between 8 and 300 GiB).
• The runtime takes any host pointer inside such an alias for the registered memory. In a bad start, hipPointerGetAttributes on a local variable returns type 1 (host registered), with hostPointer = slice host address + (variable − alias).
• The runtime reserves the CPU address range of cudaMalloc memory (VirtualQuery: MEM_RESERVE), but not the range under an alias (MEM_FREE). So a thread stack can be placed there.
• In every bad start, the alias of one slice covered the stack of the thread that pins the arena. A host-to-device cudaMemcpy from a local variable then read the arena, and a device-to-host copy into a local variable wrote into the arena. The long-prompt paths pass small values through the stack, so they compute with wrong values. Short prompts mostly do not.

This confirms the analysis that @giacomobord-prog posted in #3431.

Result: 0 bad starts in 110 starts of the final change. The same build without the change had 4 bad starts in 30. On current main (c62d5e3a1), the result is the same: 0 bad starts in 70 with this change merged in, and 5 bad starts in 70 without it.

Related discussion

• [Engine] Windows HIP (gfx1100): batched PLE block silently misreads prompts in ~1 of 8 engine starts; STRATA_PLE_BATCH=0 fixes it #3431: the root cause, a first patch as a diff, and STRATA_ARENA_PIN_GIB=none as a stopgap.
• [Engine] Windows HIP, one RX 7900 XTX: about 1 engine start in 10 answers long prompts and pictures wrong (KV streaming 0 block reads), still in 0.2.0-beta #2960: the same symptom since 0.1.44.1, and the 0 block reads marker.
• No PR for [Engine] Windows HIP (gfx1100): batched PLE block silently misreads prompts in ~1 of 8 engine starts; STRATA_PLE_BATCH=0 fixes it #3431 existed. This PR builds on the [Engine] Windows HIP (gfx1100): batched PLE block silently misreads prompts in ~1 of 8 engine starts; STRATA_PLE_BATCH=0 fixes it #3431 diff and credits it.

Approach

The change has three parts. All of them apply only to Windows HIP (#if defined(_WIN32) && defined(STRATA_USE_HIP)). CUDA, Linux HIP, SYCL and Metal keep their behavior.

1. Reserve each alias (src/core/pinned.cu, claim_alias).
   • After a range is registered, Strata reserves its alias range in the address space of the process with VirtualAlloc(MEM_RESERVE). This is address space only, nothing is committed. The runtime already does the same for cudaMalloc.
   • No later stack, heap block or mapping can then be placed there.
   • The reservations are released after the ranges are unregistered.
2. Refuse an alias that covers memory in use.
   • If any part of the alias range is not free, Strata unregisters the range again and reports it as refused. So the existing "driver refused" path applies: that range and the ranges after it stay resident but unpinned.
   • The start is then slower but correct, and the log says why. Example:
     strata pin: the GPU alias [0x4e4840000, +637 MiB) of the range at 0000016AB69E0000 covers memory in use at 0x503ce0000 (this thread's stack, from 0000000503CE0000); that range stays unregistered (#3431)
3. Keep the memory of the process above the aliases (src/program/generate.cpp).
   • main reserves the free address space below 1 TiB for the life of the process (address space only) and runs the engine on a new thread.
   • The stack of that thread, and every stack, heap block and mapping that the process allocates later, is placed above 1 TiB.
   • The aliases stay low, because the GPU allocator does not look at the CPU reservation. An alias inside the reservation is clear, because nothing can be placed there.
   • The HIP runtime moves its own cudaMalloc reservation above 1 TiB (measured: 0x10100000000 instead of 0x300000000).
   • What existed before main stays low:
     • the stack of the first thread, which only waits from then on and is exempt;
     • a few blocks of the process, which part 2 still checks.
   • STRATA_STACK_ISOLATE=0 turns part 3 off.

Part 3 is the fix that @giacomobord-prog measured in #3431 (0 bad starts in 210). Parts 1 and 2 cover two cases that the diff does not cover: an alias above 1 TiB, and memory that existed before main.

A first version of this PR released the reservation right after it created the engine thread. Then every later thread stack lay low, next to the first one, and part 2 refused a range in 7 of 60 starts. The answers were correct, but prompts in those starts were slower. Keeping the reservation lowered the refusals to 7 in 110 starts. Everything allocated after main now lies above 1 TiB, so the remaining refusals hit memory that existed before main. In the diagnostic runs of the first version, the refusals hit two kinds of memory: the first thread's stack (now exempt) and a private block next to it. The final series ran without the thread listing, so the memory of its 7 refusals is not identified.

A less strict variant would refuse only a thread stack. This PR refuses any memory in use, because it is not known which memory the runtime or a thread pool uses for small copies later.

A regression test, pinned_alias_test (Windows HIP, skips elsewhere), checks two things:

• While an arena exists, the aliases of its ranges are reserved.
• A 1 MiB buffer placed at an alias address stops the next arena from registering that range, because the runtime hands out the same alias again. A 16-byte round trip through the buffer must be correct. Without the fix, that round trip returns zeros.

A short note goes into docs/AMD_HIP.md.

Validation

Environment:

• Windows 10 19044, RX 7900 XTX (gfx1100) 24 GB, Ryzen 9 5900X, 64 GB RAM, driver 32.0.32015.7029.
• Engine built from v0.2.8 (870aedd9e) with the flags of tools/hip/build_windows.bat, CMAKE_HIP_ARCHITECTURES=gfx1100, and ROCm 10.2.0a20260930 (the ROCm of the release package). The DLLs came from the 0.1.54 rdna3 package.
• Model: Swift 1.5 IQ3_XXS with the config from setup (--kv int8 --kv-resident 32768 --spec 4 --mtp --expert-cache auto --max-context 131072, prompt switches from strata tune).

Start loop: for each start, serve/server.py ran with the config. The loop sent a 7,118-token prompt with a hidden code word (thinking off, greedy) and a short prompt, and then read the engine log of that start. A start counts as bad when the answer misses the code word. "Diag" builds add a STRATA_PIN_DIAG=1 log of every slice's alias, a 16-byte round trip through a stack variable and hipPointerGetAttributes of it. They are not part of this PR.

Build	Starts	Bad	Refused	Median prompt tok/s (good starts, full pin)
release 0.1.54 package	2	1	–	1174
v0.2.8 local build + diag	30	4	–	1177
parts 1–2 + diag	25	0	1	1179
parts 1–3, reservation released at once (diag in 139 of them)	199	0	14	1174–1181
this PR, + diag	30	0	0	1175
this PR	80	0	7	1181 (1)

(1) The 37 fully pinned starts outside a slow period (39 starts, 2 of them refused). From start 31 to 71 the whole PC was slow: a 22-token prompt took 3.4 s instead of 0.3 s. Those starts were correct and are counted above. Their speed is left out of the median. The cause of the slow period was not found.

Passed:

• The v0.2.8 local build with diag, 30 starts: in the 4 bad starts, and only in those, (a) the alias of one slice covered the pinning thread's stack, (b) the 16-byte round trip through a stack variable returned zeros, (c) hipPointerGetAttributes of a stack variable returned type 1, and (d) the log showed 0 block reads.

• The final change, 110 starts: 0 bad. The engine thread's stack was at 0x10000000000 in every diag start.

• A separate check with the same exe, 12 more starts: 0 bad, 0 refused, 36 slices pinned each time, 1116–1192 prompt tok/s.

• The 0 block reads marker identifies a bad start from its first request, also a short one:

  • In 378 starts of the loop, every request of the 5 bad starts read 0 blocks, and no request of a good start did.
  • In 39 more starts with the release package, a 22-token prompt was the first request. It read 1,176 blocks in each good start and 0 in the 2 bad starts.
  • The production log of the release package (12 starts, 3 of them bad) shows the same split.

• A standalone program (not in this PR) reproduced the fault every time. It registered a range, placed a buffer at its alias, and registered the range again. The runtime returned the same alias, and the round trip through the buffer returned zeros. The device-to-host copy wrote into the registered range instead.

• pinned_alias_test: OK. The test exercised the refusal. pin_budget_test: OK.

• CUDA compile (nvcc 13.4, MSVC 14.44, -DSTRATA_ENABLE_CUDA=ON): the objects of pinned.cu, generate.cpp and pinned_alias_test.cpp build.

• tools/strata_bench.py --config <config> --long, stock against this PR, alternating twice (each start fully pinned, 36 slices). The speed is unchanged within the spread of the runs:

  Median of 3	stock 1	stock 2	this PR 1	this PR 2
  chat decode tok/s	71.5	73.9	72.8	70.5
  agent decode tok/s	85.7	85.2	87.5	86.8
  4K prompt tok/s	836.4	827.8	841.5	831.9
  16K prompt tok/s	1793.0	1791.5	1792.9	1793.8
  32K prompt tok/s	1795.8	1790.9	1791.2	1783.4

  The time to a ready server is unchanged too: median 27.7 s (stock) and 27.7 s (this PR) in the start loop.

• Current main. The branch is based on v0.2.8 (870aedd9e). It merges into main c62d5e3a1 without conflicts. In the merged tree, all four registration sites in pinned.cu still call register_range. Two builds of main used the same flags as above: main, and main with this branch merged in. Both builds ran with the serve/server.py of main.

  • pinned_alias_test and pin_budget_test of the merged build: OK.

  • Start loop, the two builds in alternation, in blocks of 5 starts:

    Build	Starts	Bad	Refused	Median prompt tok/s (good starts, full pin)	Median ready time
    main	70	5	–	1171	28.2 s
    main + this PR	70	0	2 (2)	1170	28.2 s

    (2) At slice 16 (1078 tok/s) and at slice 2 (1048 tok/s). One more test start before the series refused slice 0. That start was correct.

  • The 5 bad starts of main look like the bad starts of v0.2.8: all 36 slices pinned, 0 block reads for every request, and fluent answers about something else.

  • tools/strata_bench.py --long of main, alternating twice (each start fully pinned, 36 slices):

    Median of 3	main 1	main 2	main + this PR 1	main + this PR 2
    chat decode tok/s	68.7	69.6	69.8	70.6
    agent decode tok/s	84.2	83.5	82.7	81.6
    4K prompt tok/s	811.3	819.3	817.7	813.7
    16K prompt tok/s	1763.9	1762.7	1763.6	1762.7
    32K prompt tok/s	1765.8	1757.2	1760.7	1754.9

Failed:

• none observed

Not run:

• A full CUDA build and CUDA execution. Linux HIP, SYCL and Metal builds. The changed code is compiled only for Windows HIP. Elsewhere only the unchanged paths compile.
• Other cards: the R9700 (gfx1201) from [Engine] Windows HIP (gfx1100): batched PLE block silently misreads prompts in ~1 of 8 engine starts; STRATA_PLE_BATCH=0 fixes it #3431, other drivers, Windows 11.
• The kept reservation of part 3 has no automatic test (pinned_alias_test has its own main). Only the start loop covers it.
• The DeepSeek engine (strata-dsv41): ds4_model.cu registers host memory too and is not changed.

Limits

• In about 5 % of starts (9 in 180), an alias still covers memory from before main. That range and the ranges after it stay unpinned. So the cost depends on the first refused slice. Measured outside the slow period, the prompts of those starts read 2–10 % slower: 1161 tok/s with a refusal at slice 30, down to 1048 tok/s at slice 2, against 1170–1181. A refusal at slice 0 leaves the whole arena unpinned. That costs as much as STRATA_ARENA_PIN_GIB=none, which made prompts 9–12 % slower on this PC. A follow-up could pin around a refused slice, but the consumers in expert_source.cpp expect a pinned prefix.
• Mapped host memory outside PinnedArena is not covered: the opt-in partial pin of FileExpertSource (STRATA_PARTIAL_PIN=1) and the DeepSeek engine. cudaHostAlloc memory has identical host and device addresses here, so it has no separate alias.
• With several GPUs, claim_alias reserves only the alias of the current device.
• A future HIP runtime that reserves the alias itself would have every range refused here, and logged.