Skip to content

fix(dsv41): run eagerly when a CUDA graph capture fails - #62

Open
nuk3s wants to merge 1 commit into
bertholomus:deepseek-v41-zig-tp2from
nuk3s:zig-tp2-graph-eager-fallback
Open

nuk3s wants to merge 1 commit into
bertholomus:deepseek-v41-zig-tp2from
nuk3s:zig-tp2-graph-eager-fallback

Conversation

@nuk3s

@nuk3s nuk3s commented Nov 19, 2026

Copy link
Copy Markdown

What this changes

With TF_DS_GRAPHS=1, one failed CUDA graph capture fails the whole round and every live request on the server. When a side fork was open at the failure, every later capture fails too, until a restart. This change makes a failed capture run the same work eagerly (kernel by kernel, as with graphs off), so the round finishes and the replies keep the same bytes. A round whose eager run fails too (a kernel variant missing from the kit, for example) still fails as before.

This is the port of #45 to deepseek-v41-zig-tp2, as requested in the review there. The review also asked for a two-node run in the served layout before merge, with replies checked token for token at the default setting. That run was not done for this port (see "Not run" below).

A capture runs nothing on the GPU, so after a failed capture nothing has run yet and the host state can be put back before the eager run.

The change has these parts:

  • zig/src/families/dsv41/graph_fallback.zig (new, 360 lines): a small Lazy(G, Key, Value) helper. The helper replaces the capture code that round.Graphs and draft.PassGraphs each had as a copy.
  • A key that fails runs eagerly for the next 1024 uses, then is captured again. Each repeat failure doubles the hold, up to 65,536. One warn line is logged per key.
  • The stuck-forever bug: a capture that failed with a fork open left side_out or pf_out set, and the next capture waited on a join event from the dropped graph, which the driver refuses. Graphs.run now joins any open fork before a replay or a capture.
  • A use-after-free: after a failed first launch, an errdefer destroyed the graph exec while the map kept the handle. The exec now stays alive in the map. A failed launch still returns the error, because the work may have started.
  • A new counter, tensorfold:graph_fallbacks_total on /metrics (rank 0, dsv41 with graphs on), documented in docs/api.md.

Every error raised under capture takes the fallback: a missing kernel variant, an instantiate error, and also a ring error.Failed or NotPortedYet. The work reruns eagerly, the warn line says a capture failed, and the counter goes up. An error that is not specific to capture returns from the eager run as before. The port tested in #45 returned pair-ending errors without an eager run; this PR does not separate those errors.

Overlap with fix/tp2-31-capture-join (a8ff91e, not in ff234ecf4): that commit ends an aborted stretch capture with the forks joined and returns the body's error instead of 904. In this PR the Lazy record path returns the body's error from the failed capture (the end-capture status is ignored), and the eager rerun's own result is what the caller sees afterwards. The PR also joins open forks at the start of Graphs.run. Both change the capture block in Graphs.run. A trial merge of the fix into this branch gives one conflict, in round.zig, and none elsewhere. Whichever lands second needs a rebase. If the fix lands first, this PR can be rebased onto the fix, and the joins can move into a new abort hook on the site.

Differences from #45, because of what this branch changed:

  • The lane host's status code moved to zig/src/core/lane_host_status.zig, so the graph_fallbacks field is read there.
  • The conflict in cuda.zig was resolved by keeping the branch's lines and adding three lines (a Host field, the assignment to the lane host, and the call to the family's graphFallbacks). The conflict in native.zig kept both sides' tests.
  • This branch added a parallel-experts stream (Round.par, par_fork, par_join, used by decodeGateUpParity). The objects are created once, and each event is recorded again before the wait on the event, inside one call. The stretch undo leaves the objects alone. This rests on reading the code: no test or run made a capture fail between the fork and the join there.
  • The warn text and the comments and test names were reworded: the line now reads "the key runs eagerly for the next N uses". The behavior is the same.

If nothing fails, the only new cost is two flag checks before each replay.

Why

MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#65 is this bug on 2x DGX Spark:

error: no captured Triton variant of _index_score for this launch: (... n: i32 200000 ... nt: i32 3125)
error: cuStreamEndCapture: CUDA_ERROR_STREAM_CAPTURE_UNJOINED (904) capturing stream has unjoined work
error: cuStreamWaitEvent: CUDA_ERROR_INVALID_VALUE (1) invalid argument
error: cuStreamEndCapture: CUDA_ERROR_STREAM_CAPTURE_INVALIDATED (901) operation failed due to a previous error during capture

In the three runs reported there, 36, 220 and 41 requests failed after that, and only a restart cleared the fault. MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#63 shows the same 901 error at about 262,144 tokens, and #51 shows cuGraphInstantiateWithFlags error 800 followed by a server that takes requests but makes no tokens. The same root cause is a guess for both: neither report shows enough to be sure. The review of #45 also names bertholomus/deepseek-v4.1-tensorfold-tp2-2xgb10#31 as a possible beneficiary.

Receipt

  • Environment: base deepseek-v41-zig-tp2 @ ff234ecf4, Zig 0.17.0, x86_64 Linux, no GPU. The GPU environment of the earlier run is in fix(dsv41): run eagerly when a CUDA graph capture fails #45.
  • Exactness: not re-run on this branch's head. In fix(dsv41): run eagerly when a CUDA graph capture fails #45 the kit's 31 exact-output cases matched a stock-image run token for token (2x DGX Spark, GB10, TP2, drafts off, with a test-only patch that made captures fail on purpose). That build was the kit's at 27741e17 (629c524a plus the kit's patches) with a port of this change. The port also restored the fork flags and ring state that the kit's KV-shard patches add, and returned the kit's pair-ending errors without an eager run, so the tested code differed from this PR.
  • Decode speed before and after: not measured. The cost of a held key and of the two flag checks was not measured either.
  • Prompt speed before and after: n/a. The prompt pass does not use graphs.
  • Tests run: zig build test --summary all, with the worktree's .zig-cache removed first: 1777 passed, 10 skipped, 0 failed. The base head gives 1766 passed, 10 skipped.
  • zig/tests/server/zig_test.sh: 403 passed, 2 skipped, 0 failed in the first suite (base 402), then 5 and 180 passed (base 5 and 179). The new server test checks the counter's line.
  • New tests: six in graph_fallback.zig drive Lazy through a fake of the graph calls (a clean capture; a failure in the work, at begin, at end capture or at instantiate; the hold cycle; the doubling; an eager error; a failed launch), plus one each in round.zig (the stretch undo), native.zig, lane_host_test.zig and metrics.zig (the counter).
  • Failing check: with the hold set to 0 after a failure (left = 0 in Holds.fail), the hold tests in graph_fallback.zig fail ("expected 1024, found 0"). The change was reverted afterwards. The other mutation checks listed in fix(dsv41): run eagerly when a CUDA graph capture fails #45 were not repeated on this branch.
  • zig build native -Doptimize=Debug: builds. This compiles the drafter's site and cuda.zig, which the host tests do not analyse.
  • zig fmt --check build.zig zig: lists the same five files on the base head and on this branch (round.zig is one of the five, with or without this change).
  • python3 tools/zig/lean_check.py: 1,863 problems on the base head and 1,863 on this branch. round.zig, draft.zig, model.zig, lane_host.zig, engine_api.zig, lane_host_test.zig and cuda.zig were already over 600 lines and grew by 62, 8, 6, 1, 2, 21 and 3 lines.
  • Measurements from fix(dsv41): run eagerly when a CUDA graph capture fails #45 (the kit build above): rank 0 logged 22 failures, each ran eagerly, and all 32 requests returned 200. These were not repeated on this branch's head.
  • Not run, and why: a two-node run on this branch's head (served layout, token-identical replies at the default setting). test-golden and the Metal build are macOS only.

Not covered:

  • The cause of the first failure, such as the missing _index_score variant or the instantiate error 800. A round that needs a kernel variant missing from the kit still fails the live streams, because the eager run fails too. The review thread on fix(dsv41): run eagerly when a CUDA graph capture fails #45 has a concrete case.
  • The drafter pass on the GPU. The run in fix(dsv41): run eagerly when a CUDA graph capture fails #45 had drafts off. Host tests cover the shared Lazy logic only.
  • NCCL collectives in an eager run after a dropped capture. The run in fix(dsv41): run eagerly when a CUDA graph capture fails #45 used the RDMA ring.
  • The joins before a replay: no run made an eager stretch fail with a fork open, so that path rests on reading the code.
  • Launch counts in the profile clocks (Round.launches, launch_mark): the launches of a failed capture and of the eager rerun are both counted. The stretch undo does not restore the counters.

Checklist

  • Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
  • Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. Not re-run on this branch's head. The run in fix(dsv41): run eagerly when a CUDA graph capture fails #45 (drafts off) matched the stock run with drafts on, and the resume and solo checks passed.
  • No precision traded for speed. The eager round launches the same kernels with the same arguments in the same order.
  • A new or changed GPU kernel is registered with the kernel contract. No kernel is added or changed.
  • Prompt processing is no slower than the last release. Not measured. The prompt pass does not use graphs.
  • The description says which platforms were run (host tests, CUDA native build) and which were not (Metal).
  • New tests fail before the change, pass after the change, and skip cleanly when the dependency is missing. See the failing check.
  • Comments and docstrings are one line. No measurements or history in the source.
  • No personal data, machine names, internal hosts or local paths. No AI attribution lines.

@bertholomus

bertholomus commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Thank you @nuk3s — the port of #45 onto deepseek-v41-zig-tp2 carries the stuck-forever and use-after-free fixes plus the fallback counter, and the honest "not run" note on the two-node run. The overlap with fix/tp2-31-capture-join (a8ff91e) is described clearly; we will settle the landing order on our side.

We are reviewing it now and will follow up in this thread. As said in the #45 review, a two-node run in the served layout with token-identical replies at the default setting — including the n=600000 case — stays the gate before any merge.

@bertholomus bertholomus added bug Something isn't working needs-decision labels Nov 19, 2026
@bertholomus

Copy link
Copy Markdown
Owner

Review done, @nuk3s — accepted. This is the invited port of #45 onto deepseek-v41-zig-tp2, and your receipts cover the build gate on this exact head — tests (1777 pass / 10 skip plus the server suite at 403/2 skip), the native build, fmt, and the new graph_fallbacks_total counter documented in docs/api.md.

As said in the #45 review, the two-node run in the served layout with token-identical replies at the default setting — including the n=600000 case — stays the gate before any merge, and that run is on our side.

One thing for you: #60 landed on deepseek-v41-zig-tp2 after you forked, and the two ports both touch zig/src/families/dsv41/native.zig and zig/src/native/cuda.zig, so this branch now conflicts with the base. A rebase onto the current base will also settle the round.zig landing order with fix/tp2-31-capture-join whichever way it goes. Thank you for the careful port.

A failed capture used to fail the round and every live stream, and a
capture that failed with a fork open left a flag set that broke every
later capture until a restart
(MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#65). A capture runs
nothing, so the stretch or pass now restores the host state and runs the
same launches eagerly. A failed key stays eager for 1024 uses, doubling
up to 65,536. Graphs.run joins an open fork first. New counter:
tensorfold:graph_fallbacks_total.
@nuk3s
nuk3s force-pushed the zig-tp2-graph-eager-fallback branch from cd48fc8 to dd3db99 Compare November 19, 2026 21:46
@nuk3s

nuk3s commented Nov 19, 2026

Copy link
Copy Markdown
Author

Rebased onto 6459b8e0b (the #60 merge). The one conflict was in zig/src/families/dsv41/native.zig, where the new test and the one from #60 sit side by side. Both stay. zig/src/native/cuda.zig merged without a conflict. git range-diff against the previous head shows the same patch, with only the hunk context moved. The head is now dd3db99b4.

On the new head:

  • zig build test: 1781 passed, 10 skipped (the base head gives 1770).
  • zig/tests/server/zig_test.sh: 403 passed and 2 skipped in the first suite, then 5 and 180 passed.
  • zig build native -Doptimize=Debug: passes.
  • zig fmt --check build.zig zig: lists the same five files as the base head.
  • python3 tools/zig/lean_check.py: 1863 problems, the same as the base head.

Not run: the two-node run in the served layout, which stays on your side as you wrote. a8ff91e (fix/tp2-31-capture-join) is still not in the base, so the overlap in Graphs.run is unchanged: the round.zig diff is identical before and after the rebase.

@bertholomus

Copy link
Copy Markdown
Owner

Rebase received, @nuk3s — thank you for re-running the receipts on the new head: 1781 pass / 10 skip, the server suites, the native build, fmt unchanged, and lean_check at 1863 = base. The two-node run in the served layout with token-identical replies at the default setting — including the n=600000 case — stays the gate before any merge, and that run remains on our side. Nothing else is owed from you on this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants