Repository navigation
Scope ContextFull admission backoff to the refused request - #55
beastllama wants to merge 1 commit into
Conversation
|
Sorry for the slow reply, and thank you for your patience. Thank you @beastllama for this one too, and for the clear scoping against #51 — the host-wide In one line: today one refused request puts a retry hold on the whole host, so later queued work that would fit stays unadmitted; this PR moves the occupancy threshold and retry deadline onto the refused job itself, keeps queue positions, and leaves the existing pool-fit fairness, single-fill, cache ownership and near-completion gates in place. The six CPU fixtures (fitting warm admission, retention and eventual admission, bounded retries, deadline priority, partial occupancy decrease, cancellation of only the waiting job) and all 61 lane-host tests passing are good evidence, and we noted your request to treat this as a draft until the platform checks are available. We are reviewing it — including the full native build on our side, since your WSL environment could not run one — and will follow up in this thread. |
|
CI never started on this head: the runs we approved on GitHub stayed queued with no jobs, and GitHub refuses to cancel or re-run them. We are closing and reopening this PR once to start fresh runs. Nothing about the PR changes. Sorry for the noise. |
|
Thank you again @beastllama. GitHub's CI never started on this head either, so we ran the same checks ourselves on |
Root cause: a host-wide
ContextFullretry hold prevents later queued work from reaching admission even when it can fit. Evidence: the CPU regression one100ee8leaves a fitting warm request unadmitted (expected 4, found nullcached tokens).Move the occupancy threshold and retry deadline onto the refused job. The existing admission picker skips jobs whose backoff is still active, preserving their queue position so an eligible head regains priority when its deadline arrives or occupancy falls. Existing pool-fit fairness, single-fill, cache ownership, and near-completion gates remain in place.
The new synchronous CPU fixtures cover fitting warm admission, retention and eventual admission of the refused job, bounded retries, deadline priority, retry after a partial occupancy decrease, and cancellation of only the waiting job. All 61 lane-host tests pass with Zig 0.17.0 in ReleaseSafe; the engine API semantic compile-only check, formatting, and diff checks also pass.
Validation limits: the broad engine API suite panics in the Imprint allocation-failure test; the same focused test also panics on the unmodified pinned base. The native build is blocked in this WSL environment by
checking cache failed: manifest_stat Unexpected, before a successful server build. GPU and end-to-end inference validation are pending. Please treat this as a draft until those platform checks are available.Related: #51 addresses fill/drain scheduling. This change addresses the separate host-wide
ContextFullretry hold.Reproduce the focused CPU checks: