Skip to content

fix(dsv41): run eagerly when a CUDA graph capture fails - #45

Closed
nuk3s wants to merge 1 commit into
bertholomus:deepseek-v41-zigfrom
nuk3s:zig-dsv41-graph-eager-fallback
Closed

nuk3s wants to merge 1 commit into
bertholomus:deepseek-v41-zigfrom
nuk3s:zig-dsv41-graph-eager-fallback

Conversation

@nuk3s

@nuk3s nuk3s commented Oct 26, 2026

Copy link
Copy Markdown

What this changes

Right now on dsv41 with TF_DS_GRAPHS=1, if one CUDA graph capture fails, the whole round fails and every live request on the server errors out. Worse, if a side fork was open when it failed, every capture after that fails too, until you restart the server. That's what three issues on the MiaAI kit look like (linked under "Why").

With this PR, a capture that fails just runs the same work eagerly (kernel by kernel, like graphs off), and the round finishes. Replies come out byte for byte the same.

Quick background if you haven't been in this code: dsv41 records each chunk of decode work (a "stretch") and each drafter pass as a CUDA graph the first time it sees that shape (its "key"), then replays the graph after that. Recording a graph doesn't run anything on the GPU. So when a capture fails, nothing has run yet, and it's safe to put the host state back and run the work directly.

What's in it:

  • New zig/src/families/dsv41/graph_fallback.zig with a small Lazy(G, Key, Value) helper. It replaces the capture code that round.Graphs and draft.PassGraphs each had their own copy of.
  • A key that fails runs eagerly for its next 1024 uses, then gets captured again. Each repeat failure doubles that, up to 65,536. It logs one warn line per key.
  • The stuck-forever bug: a capture that failed with a fork open left side_out or pf_out set, and the next capture waited on a join event from the dropped graph, which the driver refuses. Now Graphs.run joins any open fork before it replays or captures.
  • A use-after-free: after a failed first launch, an errdefer destroyed the graph exec while the map kept its handle. The exec now stays alive in the map.
  • New counter tensorfold:graph_fallbacks_total on /metrics (rank 0, dsv41 with graphs on). Added to docs/api.md.

If nothing fails, the only new cost is two flag checks before each replay.

Why

MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#65 is this exact bug on 2x DGX Spark:

error: no captured Triton variant of _index_score for this launch: (... n: i32 200000 ... nt: i32 3125)
error: cuStreamEndCapture: CUDA_ERROR_STREAM_CAPTURE_UNJOINED (904) capturing stream has unjoined work
error: cuStreamWaitEvent: CUDA_ERROR_INVALID_VALUE (1) invalid argument
error: cuStreamEndCapture: CUDA_ERROR_STREAM_CAPTURE_INVALIDATED (901) operation failed due to a previous error during capture

In their three runs, 36, 220 and 41 requests failed after that, and only a restart fixed it.

MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#63 shows the same 901 error at about 262,144 tokens, where decode needs a new bucket. MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#51 shows cuGraphInstantiateWithFlags error 800 and then a server that takes requests but never makes tokens. My guess for both is the same root cause, but neither report shows enough to be sure.

Receipt

Host tests pass on this branch (x86_64, Zig 0.17.0): zig build test 245/245 (base 236/236) and zig/tests/server/zig_test.sh 73/73 and 38/38 (base 72/72 and 37/37).

On my two DGX Sparks I ran it with a test-only patch that made captures fail on purpose. Rank 0 hit 22 failures and every one ran eagerly. All 32 requests returned 200, and the kit's 31 exact-output cases matched the stock image token for token.

Full test details and what I didn't cover

More background:

  • A stretch is a slice of a decode round, cut before each Engram layer.
  • A fork is side work that a stretch starts on another stream and joins later: the side work of TF_DS_HC_SIDE and the prefetch of TF_DS_L2_PREFETCH.
  • Rank 0 and rank 1 are the two engine processes, one per Spark, linked by the RDMA ring.
  • Before this PR, the lane host failed every live stream (failAll) when a round failed.

More on the change:

  • A capture can fail at begin, in the work, at end capture or at instantiate. In every case the stretch or pass puts its host state back and runs the same launches eagerly.
  • The eager round issues the same kernels with the same arguments in the same order as the graph. So replies keep their bytes, and both ranks make the same ring exchanges on either path.
  • A success ends a key's hold. The warn line comes at the key's first failure.
  • A stretch's undo restores the Carry, kk, has_cand, has_cblk, side_out, pf_out and the profile marks. A drafter pass keeps no host state, so its undo does nothing.
  • Graphs.run joins the open fork because an eager stretch that fails between a fork and its join leaves that work in flight. A join does nothing when its fork is closed.
  • Why the stuck bug stuck: joinSide returns on the refused wait before it clears the flag, so every later capture of a new key failed. Keys captured earlier kept replaying, because a replay doesn't touch the flags.
  • A failed graph launch still returns its error, because its work may have started. Before the fix, the next use after a failed first launch launched a destroyed graph.
  • The counter counts failed captures on rank 0. Only dsv41 with TF_DS_GRAPHS=1 shows it.

More on the kit issues:

Environment, host: x86_64 Linux, Zig 0.17.0, no GPU, this branch on fbbc9713.

Environment, GPU:

  • 2x DGX Spark (GB10), TP=2 over the RDMA ring, kernel 7.0.0-1019-nvidia, driver 580.178.04, NGC PyTorch 26.07 (CUDA 13.3).
  • Checkpoint Mia-AiLab/DeepSeek-V4.1-Flash-1017-EXL3-EAQ-v1-2.9-bpw @ 45ee5d0f, context 1,048,576, 4 streams, drafts off.
  • TF_DS_GRAPHS=1, TF_DS_HC_SIDE=1, TF_DS_L2_PREFETCH=1, the kit's KV shard on and a 64 GiB spill tier.
  • Engine: the MiaAI kit's build at 27741e17 (629c524a plus the kit's patches) with a port of this change. round.zig and draft.zig are the same at 629c524a and at fbbc9713.
  • The port also restores the fork flags and ring state that the kit's KV-shard patches add, and returns the kit's pair-ending errors without an eager run.
  • The image also had six other pending kit patches and a test-only patch. The test patch failed every 2nd instantiate and every 3rd side join inside a capture, on rank 0 only. It's not part of this PR.

Exactness: the kit's tools/exact.py ran 31 cases (prompts of 5 to 99,883 tokens, resumed prompts and 4 at once). Every token hash matches a reference run on the stock kit image with drafts on.

Decode speed and prompt speed: not measured.

Results on the GPU:

Tests run:

  • zig build test --summary all: 21/21 steps, 245/245 tests. The base gives 236/236.
  • zig/tests/server/zig_test.sh: 73/73 and 38/38, base 72/72 and 37/37. The new server test checks the counter's line.
  • Six new tests in graph_fallback.zig drive Lazy through a fake of the graph calls. One injects a failure in the work, at begin, at end capture or at instantiate, and checks that the capture ends with no graph or exec leaked, the host state comes back once, the work runs eagerly once and the key is held. The others cover a clean capture, the hold cycle, an eager error and a failed launch.
  • New tests in round.zig, native.zig and lane_host_test.zig cover the stretch's undo and the counter.
  • Mutation checks: removing the undo, the eager run, the end of the hold or the stretch's flag undo makes a new test fail. So does destroying the exec after a failed launch.
  • zig build native -Doptimize=debug: builds.
  • python3 tools/zig/lean_check.py: 1,452 problems before and after. The new file has 360 lines. round.zig (1,135 to 1,197 lines), draft.zig, model.zig and lane_host.zig were already over 600 lines and grew.
  • This branch and open PR feat(metrics): count prompt tokens restored from kept prompt states #44 apply on fbbc9713 in either order.

Not run, and why: test-golden and the Metal build, because I don't have a macOS host.

Not covered:

  • What causes the first failure, like the missing _index_score variant or the instantiate error 800. A round that needs the missing variant still fails its streams, because the eager run fails too. Other rounds capture and replay as before.
  • The drafter pass on the GPU. The GPU run had drafts off. Host tests only cover the shared Lazy logic, and the native build compiles the drafter's site.
  • The NCCL path for the decode gathers. The GPU run used the RDMA ring for them. No run tested NCCL collectives in an eager run after a dropped capture.
  • The joins before a replay. No run made an eager stretch fail with a fork open, so that path rests on reading the code.
  • Speed. I didn't measure decode or prompt speed, or the cost of a held key.

Checklist

  • Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
  • Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. On 2x DGX Spark, the run with injected failures and drafts off matches the stock run with drafts on, and its resume and solo checks pass.
  • No precision traded for speed. If the bits change, the description says which sums change and why. The eager round launches the same kernels with the same arguments.
  • Prompt processing is no slower than the last release. Not measured. The prompt pass doesn't use graphs.
  • The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency. See the mutation checks.
  • Comments and docstrings are one line. No measurements or history in the source.
  • No personal data, machine names, internal hosts or local paths. No AI attribution lines.

A failed capture used to fail the round and every live stream, and one
that failed with a fork open left a flag set that broke every later
capture until a restart (MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#65).
A capture runs nothing, so the stretch or pass now puts its host state
back and runs the same launches eagerly. A failed key stays eager for
1024 uses, doubling up to 65,536. Graphs.run joins an open fork first.
New counter: tensorfold:graph_fallbacks_total.
@bertholomus

Copy link
Copy Markdown
Owner

Sorry for the slow reply, and thank you for your patience.

Thank you @nuk3s for this carefully documented fix. In one line: with TF_DS_GRAPHS=1, one failed CUDA graph capture today fails the whole round and can leave the server failing every later capture until a restart; this PR instead puts the host state back and runs the same launches eagerly, so the round completes and replies stay byte-identical.

The receipts are strong — host tests 245/245 with mutation checks, a fault-injection run on a real pair with all 32 requests OK and the kit's exact-output cases matching token for token, and the stuck-fork and use-after-free bugs called out explicitly. We are reviewing it and will follow up in this thread.

@bertholomus

Copy link
Copy Markdown
Owner

Thank you again @nuk3s. One request: our releases now build from branch deepseek-v41-zig-tp2, not deepseek-v41-zig, and we want this fix there. Could you open a new PR against deepseek-v41-zig-tp2?

It changes the graph capture path, so before we merge we run it on two nodes in the served layout and check that replies stay token-identical with the setting at its default. It may also help tp2#31, which shows the same failure chain on v0.7.2.

@cbertucci33

Copy link
Copy Markdown

Thanks for documenting the missing-variant limit in this PR. A concrete kit-coverage case on DGX Spark (GB10, SM121) is n=600000, nt=ceil(n/64)=9375. In the published TP2 kit and TP4 kit, the matching float-score _index_score variant (HAS_BASE=1, KEYS=0, OUT=*fp32) and _score_keys variant (HAS_TMAX=1) both require nt.div16=true. Since 9375 is not divisible by 16, eager retry cannot supply either missing variant.

We compiled two additional SM121 variants from the unchanged kernel source, general in both n and nt, while retaining pointer-alignment, dtype and constexpr constraints. The tested kit preserved all 197 baseline records and all prior non-manifest payload bytes. Actual packed-AOT tests on each of two GB10 GPUs passed 122 full-output validations and 80 negative-dispatch checks, covering eager execution and CUDA graph capture/replay with changed base/visibility inputs.

These are synthetic kernel/dispatch results, not full-inference qualification. They do not resolve the initiating cause of the earlier CUDA error 1 followed by 901. This float-score case also differs from the integer-key signature quoted in the PR.

For a separate kit-generation coverage fix, would you prefer a generator/test PR against deepseek-v41-zig-tp2, or a contribution to a kit repository? Which generator and packer revisions should that contribution target?

@bertholomus

Copy link
Copy Markdown
Owner

Thank you @cbertucci33 — this is exactly the kind of concrete case the limit needed, and your caveats about what the synthetic receipts cover are fair and noted.
Your arithmetic stands: for n=600000 the tile count is 9375, which is not divisible by 16, so on the published kits the eager fallback has no matching float-score variant to run for it. The kits' own manifest fields gate these variants on tile-count divisibility; we are mapping your exact tuple now to confirm nothing else serves it.
We have added a non-divisible-by-16 size like this one to the two-node receipt we run before merging the successor PR, so the fallback's behavior on a kit gap is measured, not assumed.
On where the coverage fix belongs: there is no separate kit repository on our side — the kit ships inside the recipe repositories — so bring it as a PR against deepseek-v41-zig-tp2.
On the exact generator and packer revisions to target: we are checking our side and will reply in this thread.

@bertholomus

Copy link
Copy Markdown
Owner

Thank you for your patience, @cbertucci33 — the revisions to target, as promised in this thread.
The kit generator and packer live in the TensorFold repository on branch deepseek-v41-zig-tp2, at head c9c709534d0fa64cd62bded064e929df6f8f797d.
Under tools/dsv41-zig/kit/: aot_pack.py packs the AOT binaries, pack_kit.py copies a lane's kit and writes MANIFEST and SHA256SUMS, make_rope.py and make_bias_vl.py make the generated files, and verify_kit.sh checks the lot.
The released kits those tools made: the two-node kit v0.7.5 (engine e100ee8b, 383 files) and the four-node kit v0.3.7 (engine 4689fb4948c4, on v0.3.0's engine 5172621, 378 files).
So bring your SM121 non-div-16 variants as tests plus kit artifacts through a PR against deepseek-v41-zig-tp2.
Your n=600000 case (nt=9375) is already in the two-node token-hash receipt we run on the exact head before any merge.

@nuk3s

nuk3s commented Nov 19, 2026

Copy link
Copy Markdown
Author

Superseded by #62, the port onto deepseek-v41-zig-tp2. Closing this one.

@nuk3s nuk3s closed this Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants