feat(bench): run ROCm gates and measurement sessions under one guard lock hold #2244

Part of #1801

Problem / Background

On the shared gfx1151 host, parallel work units starved on scripts/rocm_gpu_guard.sh from 2026-10-07 to 2026-10-09:

• Units queued for hours on the host-wide lock, because the holder waits for a 60 to 90 s window with no GPU process and no compiler while other units compile and run tests.
• Unguarded make verify-rocm runs kept invalidating other guards' idle windows; test(rocm): verify the unified engine on ROCm after the batch-native engine epic #2192's parity guard was rejected 28+ times.
• A measurement script that guards each run separately takes the lock again for every run and lost it to other units each time; test(rocm): verify the unified engine on ROCm after the batch-native engine epic #2192 got 5 runs in 5 hours.

The workaround that worked: build without the lock, then run the gate or the whole measurement session inside one flock hold with ROCM_GPU_GUARD_LOCK_HELD naming the holding shell, so per-run guards inside skip the lock but keep their idle wait and monitor. It exists only as hand-written shell.

Current Behavior

scripts/rocm_gpu_guard.sh (origin/main 1b4e3657, lock from PR #2161, #2146) takes the lock in acquire_lock (:219-256) before the first idle wait and holds it until exit. A guard skips the lock only when ROCM_GPU_GUARD_LOCK_HELD names one of its ancestors (:144-152), and only a guard sets it, for its own COMMAND. There is no mode that takes the lock without the idle wait and monitor, no way to see who holds it, and flock waiters are not served in order. make verify-rocm (Makefile:806-808) runs verify-rocm-smoke and verify-test-rocm (:780-782) with no guard or lock. scripts/rocm_decode_profile.sh:122, :129 guards each run separately. docs/benchmarks.md:235-241 documents the lock but not sessions.

Proposed Solution

1. --hold mode in rocm_gpu_guard.sh: rocm_gpu_guard.sh --hold [--max-wait SECS] -- COMMAND takes the lock through the same acquire_lock (same --max-wait, exit 75 if it runs out), runs no idle wait or monitor, exports ROCM_GPU_GUARD_LOCK_HELD=$$ and runs COMMAND in the foreground, exits with COMMAND's status, and releases on exit and on INT/TERM (130/143). Under an outer guard it runs COMMAND directly. --idle-secs, --max-attempts and --log are rejected with --hold (exit 2).
2. make verify-rocm-held: first $(CARGO) test --workspace --profile test-fast $(ROCM_JOBS) --features rocm --no-run and the smoke build with no lock, then scripts/rocm_gpu_guard.sh --hold -- $(MAKE) verify-rocm. Rebuilds inside the hold are then cache hits.
3. --status: prints whether the lock is held and, from a sidecar $LOCK.holder written at acquire (pid, start time, command line; created like the lock file, world-writable), the holder's details; a stale sidecar whose pid is gone is reported as stale. Exit 0 free, 1 held.
4. Docs: a "Sessions on a shared host" paragraph under "ROCm hosts" in docs/benchmarks.md: build unlocked, run gates with make verify-rocm-held, wrap multi-run measurements in one --hold with per-run guards inside, never run make verify-rocm unheld on a shared host.

FIFO fairness (a ticket queue in front of flock) is out of scope unless --hold sessions still starve after this lands; file it separately with measurements then.

Acceptance Criteria

• New GuardTests cases in tests/test_rocm_decode_profile.py: a --hold session runs two nested guards that take no lock (their logs show lock held by outer guard); a second --hold waits while the first holds and exits 75 on --max-wait; --status reports held then free; --hold --idle-secs 5 exits 2.
• make verify-rocm-held runs to completion on the gfx1151 host, and a concurrent rocm_gpu_guard.sh -- true waits for it rather than being rejected.
• docs/benchmarks.md documents the session workflow.

Verification

python3 -m unittest tests.test_rocm_decode_profile -v
make verify-python-tooling
make verify-rocm-held

Related: #2146, PR #2161, #2192, #2061.