feat(pflash): auto skip-park residency with VRAM estimate and OOM fallback

[Graffioh]

Summary

Stacked on #746. The branch merges #746's current head (4d011e68f) and then main (85a25c4f0). The commits specific to this PR are dc219e6f (the feature) and 847c26e3 (review fixes). Once #746 merges, the diff shrinks to those two commits.

--prefill-skip-park becomes tri-state auto|on|off. The default is auto, and the bare flag still means on. Parking the target and decode draft around every compression window adds a flat ~4 s to TTFT on each long prompt. When VRAM is ample, that round-trip does no useful work.

What it does

• Startup estimate. resolve_skip_park in placement/skip_park_guard.h runs once in server_main.cpp, after the backend has loaded. inspect_drafter_footprint reads the drafter's GGUF header and follows the buffers the Qwen3.5 strict scorer allocates:

  • PFLASH_DRAFTER_SESSIONS scoring sessions (default 2). Each is sized S + S/2 + 4096 and holds the KV of blocks 0..14 plus f32 block-15 keys.
  • Activations.
  • The [S, nq, n_head] logits of the scorer query window. nq comes from PFLASH_SELECT_QUERY_TOKENS and can be up to 512.

  The estimate is taken over min(max_ctx, drafter ctx). Skip-park turns on when the footprint plus a 25% margin fits the free VRAM measured on the drafter device. The estimate honors LUCE_KV_*. TQ3 is suppressed, as it is in the drafter's own cache.

• Keep-loaded (auto only). With --draft-residency auto, the drafter stays loaded between requests only when the auto estimate also fits the drafter resident next to the target's compute reserve. That reserve is 1.5 GiB, the same one the KV-pool budgets use for runtime graph buffers. The check is (window footprint + 1.5 GiB) × 1.25 ≤ free. The window footprint counts everything the drafter keeps between requests: its weights, its kept sessions and its compute pool. An explicit on runs no estimate, so it never changes the drafter's residency.

• Precedence. off wins over on, and on wins over auto. Remote-IPC and upstream-proxy drafters always resolve to off.

• VMM guard, now scoped. The <32 GiB && max_ctx > 64K → park guard (1c562eb) protects against CUDA's VMM pool: dual residency fragmented its address space and cuMemSetAccess failed on a 24 GB card at 128K. It now applies only when the GPU backend reports a VMM pool (its ggml_backend_get_features list has no NO_VMM). HIP builds default to GGML_HIP_NO_VMM=ON, so on the R9700 (34.2 GB = 31.86 GiB, just under the line) the estimate decides instead of the guard. CUDA builds with the VMM pool keep the guard, including for explicit on.

• Fail-safe, OOM only. The qwen35, deepseek4 and qwen3 compress paths share run_skip_park_window().

  • Only a no-park window that ran out of device memory is retried with parking. Out-of-memory covers failures to allocate the drafter's weights, cache, sessions, activations or graphs, and computes that return GGML_STATUS_ALLOC_FAILED. These now report through set_last_oom_error() and CompressResult::out_of_memory.
  • Every other failure is returned unchanged. For example, non-finite Qwen3.5 scoring-head scores does not trigger a pointless parked retry.
  • After an OOM recovers, parking is no longer latched for the server's lifetime. The next 4 windows park, doubling on each repeat up to 64, and then skip-park is tried again. A clean no-park window resets the backoff.
  • While an OOM is unresolved, the drafter is released after each window even under keep-loaded.

• Observability. A startup log line records the mode, the decision and its reason, the window and keep-loaded GiB, free GiB, the window and query tokens, and vmm_pool. /props.pflash gains skip_park_mode, skip_park_estimate_bytes, skip_park_free_bytes and drafter_keep_loaded. The disabled-PFlash /props body now includes every field the schema requires.

Evidence so far

Frozen gate binary, --prefill-skip-park forced on, Qwen3.5-0.8B drafter, R9700 (gfx1201):

run	parked	skip-park	notes
dev-qa-30 @32k, median TTFT	11.7 s	7.5 s (−36%)	identical selections and answers (26/30 both); peak VRAM 20.1 of 34.2 GB
single 120K smoke, TTFT	59.3 s	55.2 s

The saving is a flat ~4 s per request. It does not include the keep-loaded gain: in those runs the drafter still reloaded on every request (31 loads for 31 requests).

GPU validation (scheduled)

The PR build is queued on the R9700 behind the multi-turn runs, which have GPU-0 priority today. The script is .benchmark-work/pr747/gated-validation.sh and it runs:

• dev-qa-30 @32k with auto, off and on: TTFT, answer and selection equality, the startup decision, drafter loads per request, and peak VRAM sampled every 2 s.
• 5 niah cases @120000 and 4 code-debug cases @98304 with on forced (no guard on HIP), auto and off: whether anything OOMs, peak VRAM, TTFT.

This description will be updated with the numbers.

Remaining risks

• The target-prefill reserve for keep-loaded is the codebase's 1.5 GiB convention, not a value measured per target. The scheduled runs record peak VRAM during target prefill with the drafter resident, to check it.
• The fallback only catches failures the code reports. Compute-pool allocations inside ggml-cuda/hip (FA/cuBLAS scratch) abort the process instead of returning an error, so an estimate that is too low can still crash rather than fall back.
• The guard stays active on CUDA VMM builds. Whether a CUDA build with GGML_CUDA_NO_VMM and ≥ 24 GB is safe at >64K without the guard has not been tested.

Test plan

• test_server_unit (HIP gfx1201 build): 707 passed, 0 failed. New cases cover:
  • the guard applies only with a VMM pool (R9700 total VRAM)
  • keep-loaded comes only from auto
  • the estimator on synthetic Qwen3.5 GGUF headers: query-window term, sessions, defaults, non-Qwen3.5 rejected
  • a non-OOM failure does not retry
  • an OOM retries once with parking
  • the park backoff re-probes, doubles, caps and resets
• R9700 validation above (scheduled)

Generated with Devin, review fixes with Claude Code

🤖 Generated with Claude Code