Repository navigation
Conversation
A failed capture used to fail the round and every live stream, and one that failed with a fork open left a flag set that broke every later capture until a restart (MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#65). A capture runs nothing, so the stretch or pass now puts its host state back and runs the same launches eagerly. A failed key stays eager for 1024 uses, doubling up to 65,536. Graphs.run joins an open fork first. New counter: tensorfold:graph_fallbacks_total.
|
Sorry for the slow reply, and thank you for your patience. Thank you @nuk3s for this carefully documented fix. In one line: with The receipts are strong — host tests 245/245 with mutation checks, a fault-injection run on a real pair with all 32 requests OK and the kit's exact-output cases matching token for token, and the stuck-fork and use-after-free bugs called out explicitly. We are reviewing it and will follow up in this thread. |
|
Thank you again @nuk3s. One request: our releases now build from branch It changes the graph capture path, so before we merge we run it on two nodes in the served layout and check that replies stay token-identical with the setting at its default. It may also help tp2#31, which shows the same failure chain on v0.7.2. |
|
Thanks for documenting the missing-variant limit in this PR. A concrete kit-coverage case on DGX Spark (GB10, SM121) is We compiled two additional SM121 variants from the unchanged kernel source, general in both These are synthetic kernel/dispatch results, not full-inference qualification. They do not resolve the initiating cause of the earlier CUDA error 1 followed by 901. This float-score case also differs from the integer-key signature quoted in the PR. For a separate kit-generation coverage fix, would you prefer a generator/test PR against |
|
Thank you @cbertucci33 — this is exactly the kind of concrete case the limit needed, and your caveats about what the synthetic receipts cover are fair and noted. |
|
Thank you for your patience, @cbertucci33 — the revisions to target, as promised in this thread. |
|
Superseded by #62, the port onto |
What this changes
Right now on dsv41 with
TF_DS_GRAPHS=1, if one CUDA graph capture fails, the whole round fails and every live request on the server errors out. Worse, if a side fork was open when it failed, every capture after that fails too, until you restart the server. That's what three issues on the MiaAI kit look like (linked under "Why").With this PR, a capture that fails just runs the same work eagerly (kernel by kernel, like graphs off), and the round finishes. Replies come out byte for byte the same.
Quick background if you haven't been in this code: dsv41 records each chunk of decode work (a "stretch") and each drafter pass as a CUDA graph the first time it sees that shape (its "key"), then replays the graph after that. Recording a graph doesn't run anything on the GPU. So when a capture fails, nothing has run yet, and it's safe to put the host state back and run the work directly.
What's in it:
zig/src/families/dsv41/graph_fallback.zigwith a smallLazy(G, Key, Value)helper. It replaces the capture code thatround.Graphsanddraft.PassGraphseach had their own copy of.warnline per key.side_outorpf_outset, and the next capture waited on a join event from the dropped graph, which the driver refuses. NowGraphs.runjoins any open fork before it replays or captures.errdeferdestroyed the graph exec while the map kept its handle. The exec now stays alive in the map.tensorfold:graph_fallbacks_totalon/metrics(rank 0, dsv41 with graphs on). Added todocs/api.md.If nothing fails, the only new cost is two flag checks before each replay.
Why
MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#65 is this exact bug on 2x DGX Spark:
In their three runs, 36, 220 and 41 requests failed after that, and only a restart fixed it.
MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#63 shows the same 901 error at about 262,144 tokens, where decode needs a new bucket. MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#51 shows
cuGraphInstantiateWithFlagserror 800 and then a server that takes requests but never makes tokens. My guess for both is the same root cause, but neither report shows enough to be sure.Receipt
Host tests pass on this branch (x86_64, Zig 0.17.0):
zig build test245/245 (base 236/236) andzig/tests/server/zig_test.sh73/73 and 38/38 (base 72/72 and 37/37).On my two DGX Sparks I ran it with a test-only patch that made captures fail on purpose. Rank 0 hit 22 failures and every one ran eagerly. All 32 requests returned 200, and the kit's 31 exact-output cases matched the stock image token for token.
Full test details and what I didn't cover
More background:
TF_DS_HC_SIDEand the prefetch ofTF_DS_L2_PREFETCH.failAll) when a round failed.More on the change:
warnline comes at the key's first failure.kk,has_cand,has_cblk,side_out,pf_outand the profile marks. A drafter pass keeps no host state, so its undo does nothing.Graphs.runjoins the open fork because an eager stretch that fails between a fork and its join leaves that work in flight. A join does nothing when its fork is closed.joinSidereturns on the refused wait before it clears the flag, so every later capture of a new key failed. Keys captured earlier kept replaying, because a replay doesn't touch the flags.TF_DS_GRAPHS=1shows it.More on the kit issues:
cuGraphInstantiateWithFlags: CUDA_ERROR_NOT_PERMITTED (800)— chat hangs, only restart recovers MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#51, my guess is that only the failing rank gave up the round, and its peer sat waiting in the ring.Environment, host: x86_64 Linux, Zig 0.17.0, no GPU, this branch on
fbbc9713.Environment, GPU:
TF_DS_GRAPHS=1,TF_DS_HC_SIDE=1,TF_DS_L2_PREFETCH=1, the kit's KV shard on and a 64 GiB spill tier.629c524aplus the kit's patches) with a port of this change.round.ziganddraft.zigare the same at629c524aand atfbbc9713.Exactness: the kit's
tools/exact.pyran 31 cases (prompts of 5 to 99,883 tokens, resumed prompts and 4 at once). Every token hash matches a reference run on the stock kit image with drafts on.Decode speed and prompt speed: not measured.
Results on the GPU:
graph_fallbacks_totalread 22 on rank 0. Rank 1 logged no failure.finish=stop.cuStreamWaitEventerror and no 901.cuGraphInstantiateWithFlags: CUDA_ERROR_NOT_PERMITTED (800)— chat hangs, only restart recovers MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#51 hit without the fallback.Tests run:
zig build test --summary all: 21/21 steps, 245/245 tests. The base gives 236/236.zig/tests/server/zig_test.sh: 73/73 and 38/38, base 72/72 and 37/37. The new server test checks the counter's line.graph_fallback.zigdriveLazythrough a fake of the graph calls. One injects a failure in the work, at begin, at end capture or at instantiate, and checks that the capture ends with no graph or exec leaked, the host state comes back once, the work runs eagerly once and the key is held. The others cover a clean capture, the hold cycle, an eager error and a failed launch.round.zig,native.zigandlane_host_test.zigcover the stretch's undo and the counter.zig build native -Doptimize=debug: builds.python3 tools/zig/lean_check.py: 1,452 problems before and after. The new file has 360 lines.round.zig(1,135 to 1,197 lines),draft.zig,model.zigandlane_host.zigwere already over 600 lines and grew.fbbc9713in either order.Not run, and why:
test-goldenand the Metal build, because I don't have a macOS host.Not covered:
_index_scorevariant or the instantiate error 800. A round that needs the missing variant still fails its streams, because the eager run fails too. Other rounds capture and replay as before.Lazylogic, and the native build compiles the drafter's site.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. On 2x DGX Spark, the run with injected failures and drafts off matches the stock run with drafts on, and its resume and solo checks pass.