ci(e2e): run Qwen and DS4 end to end on lucebox3

[Graffioh]

What

A new Model e2e workflow (.github/workflows/model-e2e.yml) runs a real model on lucebox3 and checks whether a change crashes it or changes its behaviour.

• Qwen3.8-27B (IQ4_XS + dflash2 draft) runs on the R9700 (gfx1201).
• DeepSeek V4 Flash (STRIX mix, sparse prefill) runs on the Strix Halo (gfx1151).

Each job builds luce_server for its GPU, loads the model, sends 16 fixed prompts with greedy decoding (.github/ci/e2e/prompts.json), stops the server, and compares with the last good run on main (the baseline). The prompts cover arithmetic, lists, code, JSON, translation, unicode, multi-turn, a tool call, a 2.5K-token needle, streaming, thinking, a longer story, and a repeated request.

Result	When
❌ fail	crash · hang · server won't load or stop · GPU errors in the kernel log · a request fails or times out · the suite runs out of time · a model or the draft is missing (a PR skips instead) · 2+ checks that passed on the baseline now fail · under half the checks pass
⚠️ warn	output text differs from the baseline (with the character where it first differs) · a single regressed check · a repeated request answers differently · decode speed −15% · load time +50% · the kernel log can't be read, or lost messages during the run

When it runs

• Only when a maintainer adds the e2e label, like compact-commits in ci(commits): check conventional commit style and dispatch the compaction bot #755. It runs the models the PR touches (select_models.py), or both when the PR touches neither DS4, Qwen nor shared server code. To test new pushes, remove the label and add it again.
• Nightly on main: refreshes the baseline. Manual dispatch refreshes the baseline when update_baseline is ticked on main.

lucebox3 only

• Routing: jobs use runs-on: [self-hosted, lucebox3], next to gpu-tests-amd. The runner runs one job at a time, so e2e jobs never share the GPUs with each other or with gpu-tests-amd. Spreading jobs over more luceboxes can come later.
• One table per model: select_models.py holds each model's GPU, HIP index (R9700 = 0, Strix Halo = 1, as in gpu-tests-amd), model files and server flags. The job reads them from its matrix. Models live in vars.LUCEBOX_MODELS_DIR, default /opt/models.
• Baseline: baseline runs on main upload their result as the artifact model-e2e-baseline-<model>-<device> (kept 90 days). find_baseline.py gives each job the newest one, counting only scheduled or dispatched model-e2e runs on this repo's main, so PR code can't replace it. Results record the ROCm version, and the report warns when the baseline ran on a different one.
• Someone using the GPU by hand: the job waits up to 4 minutes, then skips with a warning. To keep jobs off while you benchmark, stop lucebox3's runner service.
• Seeing every user's GPU processes: the job reads the kernel's KFD process list (/sys/class/kfd/kfd/proc, readable by any user; it's what rocm-smi --showpids uses) and fails if it can't.

Staying bounded

• 35-minute job cap, plus caps per step: build 12 min, suite 15 min, model load 480 s (DS4 takes ~185 s from a cold page cache, ~30 s warm), 120 s per request, 300 s for the whole suite.
• Skips repeats: a clean pass (not a warning, which must show again) is remembered by tree hash, configuration, model files, ROCm version and the baseline's digest. The suite is judged against a snapshot of the baseline taken with the key, so a new baseline always re-runs it. A skipped run shows the report of the run it reused.
• Builds incrementally: the build folder persists on lucebox3 and uses ccache if installed.

Queue fix (separate commit)

By default a concurrency group keeps only one waiting job, so a second PR's waiting GPU job cancelled the first one's. queue: max (GitHub, May 2026) keeps up to 100 waiting, first in first out, on gpu-tests, gpu-tests-amd and speed-profile. actionlint doesn't know the key yet, but GitHub's docs confirm it at job level with cancel-in-progress: false.

Measured on lucebox5 (same R9700 + Strix Halo hardware)

	cold build	suite incl. load	load	decode
Qwen3.8-27B IQ4_XS + dflash2	100 s	~30 s	6–18 s	~105 tok/s
DS4 STRIX mix, sparse prefill	95 s	~80 s	28–31 s warm, 185 s cold	~19 tok/s

• Deterministic: both models gave identical text on all 16 prompts across two runs.
• Workflow steps replayed: I ran the job's shell steps on lucebox5 with Qwen: persistent build, suite (16/16), pass marker, a cached skip, a re-run once a baseline appeared (same text on all prompts), and writing the new baseline. That was before the switch to lucebox3 only, which dropped the GPU-index lookup and model search from those steps. gpu_wait.sh reports busy when another process holds /dev/kfd. find_baseline.py ran against the live API; the artifact download itself can only run once a baseline exists on main.
• Why DS4 uses sparse prefill: exact prefill runs at about 21 tok/s on the Strix Halo, and the 2.5K-token needle alone took 120 s.
• DS4 arithmetic, for reviewers: exact prefill says 60/1.5 = 10, sparse says 17×23 = 340. Both are deterministic, and Qwen gets both right. That's why a single regressed check only warns. It may be a real accuracy issue worth checking separately.

Before enabling

• Put the three models in /opt/models on lucebox3 (or set LUCEBOX_MODELS_DIR): Qwen3.8-27B-UD-IQ4_XS.gguf, qwen38-dflash2-q8_0.gguf, DeepSeek-V4-Flash-0731-ROCMFPX-MIX-STRIX.gguf.
• Make the kernel log readable for the runner user on lucebox3: passwordless sudo dmesg or kernel.dmesg_restrict=0. Without it every run warns that GPU errors weren't checked.
• After merging, dispatch the workflow on main with update_baseline ticked to create the first baseline (or wait for the nightly run).
• Create the e2e label.
• New kernel-log lines are matched by their boot timestamp, not by line count, so ring-buffer rollover can't hide a fault. When the log after the run no longer reaches back to the messages from before it, the report says so and warns.

🤖 Generated with Claude Code

[davide221]

The commit compaction bot stopped.

The pull request is still a draft.

Maintainers: see the lucebox_bots run log.

[davide221]

The commit compaction bot stopped.

Pushing to Graffioh/lucebox-hub failed (the branch moved, the fork blocks maintainer edits, or the token lacks the workflow scope). The branch is unchanged.

Maintainers: see the lucebox_bots run log.

[davide221]

Folded 2 commits into 2 (same final code; the tree hash was checked before pushing).

New commit	From
158ac7ab ci(workflows): queue GPU jobs instead of replacing waiting ones	341c98c
7ae026fb ci(e2e): run Qwen and DS4 end to end on any free lucebox	437f06c

Previous head: 437f06c2ce7c5c7b036de33f0e0d43782cf7495c.

[davide221]

Folded 3 commits into 2 (same final code; the tree hash was checked before pushing).

New commit	From
2b7e938d ci(workflows): queue GPU jobs instead of replacing waiting ones	875bd28
4eff4ad8 ci(e2e): run Qwen and DS4 end to end on any free lucebox	09f3658, 1121a5e

Previous head: 1121a5e57bd44975931025e6812e143b505e5347.

[cubic-dev-ai]

All reported issues were addressed across 12 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>