Repository navigation
perf(lanes): TF_FILL_DRAIN=ROWS starts a burst of short prompts in one turn, +56-60% at 8 streams under TF_FILL_MANY (off by default) - #51
Conversation
…n, +56-60% at 8 streams under TF_FILL_MANY With TF_FILL_CHUNKS and TF_FILL_MANY, run() takes one fillOne() a turn and a round follows it, so a burst of short prompts starts a round apart: P1 R P2 R P3 R P4 instead of P1 P2 P3 P4 R. TF_FILL_DRAIN=ROWS lets the lane host take passes that go in whole at their first step back to back, re-admitting between them, until a pass needs pieces, the lanes are full, or the next pass's rows would pass ROWS (the first always runs). Passes in pieces keep TF_FILL_ROUNDS. Unset or 0, Info.fill_drain is 0 and run() takes its current path. Two lane host tests: a burst without it starts a round apart; with it, on the stepped and the one-pass backends, it starts before the next round, up to its row budget.
|
Sorry for the slow reply, and thank you for your patience. Thank you @OsakaTX for this measured and honest addition. In one line: under The receipts are solid: five passes with alternating switch orders, 78/78 |
|
CI never started on this head: the runs we approved on GitHub stayed queued with no jobs, and GitHub refuses to cancel or re-run them. We are closing and reopening this PR once to start fresh runs. Nothing about the PR changes. Sorry for the noise. |
|
Merged on branch GitHub's CI never started on this head, so we ran the same checks ourselves on
|
What this changes
With
TF_FILL_CHUNKSandTF_FILL_MANY, the lane host starts one prompt pass a turn and a decode round follows each,so short prompts that arrive together start a round apart: P1 R P2 R P3 R P4 instead of P1 P2 P3 P4 R. On two GB10s
with 8 lanes, that costs a burst of 8 short requests 36-38% of its aggregate decode rate against the same build with
the fill switches unset.
TF_FILL_DRAIN=ROWSlets the host start passes that go in whole back to back in the same turn, up to ROWS rows. UnderTF_FILL_MANYit lifts that burst's aggregate rate by 56-60% (code: 198 to 318 tok/s; chat: 185 to 288) and bringsits slowest first token from 1.01 s to 0.60 s. That is the unset numbers again, not past them. Passes in pieces keep
TF_FILL_ROUNDS. Unset or 0, nothing changes.In detail, with
TF_FILL_DRAIN=ROWS(andTF_FILL_CHUNKS+TF_FILL_MANY):run()keeps callingfillOne()in the same turn, re-admitting between calls, while the next pass goes in wholeat its first step.
rows past ROWS. The first pass always runs.
Info.fill_drainis 0 andrun()takes its current path.Files:
core/engine_api.zig(Info.fill_drain),families/dsv41/native.zig(readsTF_FILL_DRAIN),native/cuda.zig(passes it through),core/lane_host.zig(fillOne()/run()), andcore/lane_host_fill_test.zig(two tests). I leftTF_FILL_DRAINout of thenative.zigheader table, because athree-line entry there adds lean findings. It is documented on one line where it is read.
Receipt
Environment: two GB10 nodes, TP2, the RDMA ring. Ubuntu 24.04.5, kernel 7.0.0-1019-nvidia, driver 595.99.02,
nvcr.io/nvidia/pytorch@sha256:2140e699b3beaf7f96a0081fd9c9406bc3832b435cdb60dfa2d261f7d2f34a1c.afc7aa0a, with awo_bweight patch inblocks 2-39, the same base as feat(lanes): TF_FILL_CUT_IN=T lets a short prompt run between a long prompt pass's pieces (off by default) #43); engram from
deepseek-ai/DeepSeek-V4.1-Flash-1017at1bae0e10.deepseek-v41-zig-tp2ata57deb9dplus this change, plus an env switch for the RDMA GID index (ourRoCE v2 GID is 3). Its code lines are identical to this PR's; only comment wording differs.
--parallel 8,--prompt-cache-gib 0. Engine env in everyarm:
TF_DS_GRAPHS=1 TF_DS_HC_SIDE=1 TF_DS_L2_PREFETCH=1 TF_DS_ENGRAM_AIO=1 TF_DS_REPLAY_FLOOR=1 TF_DS_IDX_SPLIT=1 TENSORFOLD_PLAIN_SPECIALS=1 TF_DS_PROMPT_ALIAS=1 TF_DS_SOLO_OVERHEAD=1 TF_DS_NO_COPIES=1, plus the grammarlibrary, SSE keepalive and stats settings.
TF_FILL_*switches differed between arms; each start's container env was read back and checked.Concurrency:
tools/bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --levels 1,4,8 --alone --serial,run in the order DRAIN, MANY, unset, DRAIN, MANY (two passes of each switch, alternated). Medians, tok/s:
CHUNKS=1 MANY=1CHUNKS=1 MANY=1 DRAIN=4096Exactness: in all five passes, every concurrent reply's
token_sha256equals its alone run's (78/78 per pass),and the alone runs equal
"draft": false(2/2 per pass).Decode speed (
tools/bench_openai.py --tokens 256 --reps 3 --temperatures 0, fibonacci-raw / gpu-chat-no-think):unset 79.44 / 69.50, MANY 79.42 / 69.53 and 79.43 / 69.51, DRAIN 79.83 / 69.68 and 79.54 / 69.61.
What it does not do: with the fill switches unset, short prompts in a burst already start together. DRAIN brings
TF_FILL_MANYback to that level, not past it. Its use is to runTF_FILL_MANY(for long prompts beside livestreams) without paying for it on bursts of short ones. If you'd rather not have another switch, it could become
TF_FILL_MANY's behaviour with a fixed budget; happy to change it.New tests (
core/lane_host_fill_test.zig, over the fake backend): three 4-token prompts all wait before the hoststarts, and a hook records the core's round count at each one-step pass.
fill_drain, the three passes start at increasing round counts (a round apart).fill_drain = 64, all three start before the first round, on the stepped backend and on the one-pass-at-a-timebackend.
fill_drain = 8, the first two start together and the third waits for a later round.Mutation checks (break one line, run
zig build test, restore):fillOne()never asking for another pass (return falseforreturn h.info_.fill_drain > 0): thefill_draintest fails.
fill_draintest fails.job.fill_begandropped from the drain check: not caught, and I don't think a test can catch it. Once a wholepass has gone in, its stream is live and
fill_sinceis 0, so theTF_FILL_ROUNDScheck on the next line alreadyholds back a pass that began. The term changes something only with
TF_FILL_ROUNDS=0: there, a begun pass's nextpiece runs before the round instead of after it, and that setting allows either. I kept it to state the intent, so
the code matches what we measured. Dropping it is fine with me.
Tests run (x86_64 Linux, Zig 0.17.0, CPU only):
zig build test: 1766/1776 passed, 10 skipped (1762/1772 ona57deb9d, with the same 10 skipped). The skipsare the same five binaries in both trees, each printing
libjpeg-turbo 3.1.4.1 not found.python3 tools/zig/lean_check.py+tools/zig/lean_new.py: 1855 problems, 1855 on the base, 0 new.zig fmt --checkon the five files is clean.