Feat/laned runtime - #2765
Open
kali wants to merge 7 commits into
Open
Conversation
A laned runtime hands a lane to each stream it serves and seats some of them in every turn. LaneTable is that bookkeeping, on its own: plain data, no model and no device, so clearing what a stream left behind stays with the caller owning the state.
LanedRunnable spawns a handle per stream, each holding a lane of one long-lived state, and a worker thread owning the state and the lane table both runs the turns queued at any moment as one: axis 0 of the batched tensors carries a row per seat, and each stream is handed back its own. Turn policy through TRACT_MAX_SEATS and TRACT_TURN_LINGER_US.
A laned runnable resets every lane of its state once, at prepare: an op state that cannot serve several streams at once fails there, rather than at whichever turn first seats a second stream.
Mean occupancy is what says whether the queue is offering wide turns, and it is the dial TRACT_MAX_SEATS and TRACT_TURN_LINGER_US are set against.
The memo held one set of stores per node, and their strides come from the output shape: a turn whose batch axis has another extent than the first one's read the wrong layout. Rebuild them when the shape or the kernel changes.
The staggered-seats case seats lanes by hand. Add one where a pulsed pad plus conv is served by a LanedRunnable with a stream per thread, so the seating is whatever the worker finds queued, and every stream still gets what it gets running alone.
Its counter advances once per turn of the state, which is a stream's own turn count only while every turn seats every stream.
|
🔴 Bench vs main — 2 speed regression(s) Reference: 2026-09-02 morning nightly run (0d old) · full report → run Speed — evaltime · prefill · decode
Improvements
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.