Skip to content

fix(vllm-4b): default to vLLM 0.23 on ROCm to stop the idle HSA spin - #390

Open
kh0pper wants to merge 1 commit into
mainfrom
fix/vllm-4b-image-idle-spin
Open

kh0pper wants to merge 1 commit into
mainfrom
fix/vllm-4b-image-idle-spin

Conversation

@kh0pper

@kh0pper kh0pper commented Sep 29, 2026

Copy link
Copy Markdown
Owner

The 4b classifier bundle defaulted to rocm/vllm-dev:nightly_main_20260413 (vLLM 0.19.1rc). On that build, the first request leaves a native thread in VLLM::EngineCore spinning at 100% of a CPU core for as long as the container runs: the ROCm HSA runtime's event thread, whose AMDKFD_IOC_WAIT_EVENTS wait keeps returning immediately. This is vllm-project/vllm#41964, fixed in vLLM 0.21.0.

Change: the compose default moves to AMD's rocm/vllm:rocm7.14.1_rdna_ubuntu24.04_py3.14_pytorch_2.11_vllm_0.23.0, which is validated on gfx1151, and a two-line comment records why. A host that sets VLLM_ROCM_IMAGE in the bundle .env still overrides it.

Diagnosis on a gfx1151 host:

  • py-spy showed every Python thread idle.
  • gdb put the hot thread in libhsa-runtime64.
  • A fresh container stayed idle until the first request, then spun.

Before/after on the same host, same checks (idle GPU):

vLLM 0.19.1rc nightly vLLM 0.23.0
latency p50 / p90, 20 short requests 0.201 / 0.203 s 0.181 / 0.182 s
answers on 5 fixed prompts reference identical
qwen3_xml tool call structured get_weather call identical
hottest engine thread 60 s after requests 99.9% 0.0%

Also checked:

  • Every flag the compose passes is accepted by the new image.
  • qwen3_xml is registered.
  • 40 more requests showed no latency outliers (max 0.183 s).
  • node --test tests/bundle-contract.test.js: 28/28 pass.

The new image is about 75 GB on disk, against about 11 GB for the old one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant