Repository navigation
Conversation
|
Thank you @nuk3s — the port of #45 onto We are reviewing it now and will follow up in this thread. As said in the #45 review, a two-node run in the served layout with token-identical replies at the default setting — including the n=600000 case — stays the gate before any merge. |
|
Review done, @nuk3s — accepted. This is the invited port of #45 onto As said in the #45 review, the two-node run in the served layout with token-identical replies at the default setting — including the n=600000 case — stays the gate before any merge, and that run is on our side. One thing for you: #60 landed on |
A failed capture used to fail the round and every live stream, and a capture that failed with a fork open left a flag set that broke every later capture until a restart (MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#65). A capture runs nothing, so the stretch or pass now restores the host state and runs the same launches eagerly. A failed key stays eager for 1024 uses, doubling up to 65,536. Graphs.run joins an open fork first. New counter: tensorfold:graph_fallbacks_total.
cd48fc8 to
dd3db99
Compare
|
Rebased onto On the new head:
Not run: the two-node run in the served layout, which stays on your side as you wrote. |
|
Rebase received, @nuk3s — thank you for re-running the receipts on the new head: 1781 pass / 10 skip, the server suites, the native build, fmt unchanged, and lean_check at 1863 = base. The two-node run in the served layout with token-identical replies at the default setting — including the n=600000 case — stays the gate before any merge, and that run remains on our side. Nothing else is owed from you on this PR. |
What this changes
With
TF_DS_GRAPHS=1, one failed CUDA graph capture fails the whole round and every live request on the server. When a side fork was open at the failure, every later capture fails too, until a restart. This change makes a failed capture run the same work eagerly (kernel by kernel, as with graphs off), so the round finishes and the replies keep the same bytes. A round whose eager run fails too (a kernel variant missing from the kit, for example) still fails as before.This is the port of #45 to
deepseek-v41-zig-tp2, as requested in the review there. The review also asked for a two-node run in the served layout before merge, with replies checked token for token at the default setting. That run was not done for this port (see "Not run" below).A capture runs nothing on the GPU, so after a failed capture nothing has run yet and the host state can be put back before the eager run.
The change has these parts:
zig/src/families/dsv41/graph_fallback.zig(new, 360 lines): a smallLazy(G, Key, Value)helper. The helper replaces the capture code thatround.Graphsanddraft.PassGraphseach had as a copy.warnline is logged per key.side_outorpf_outset, and the next capture waited on a join event from the dropped graph, which the driver refuses.Graphs.runnow joins any open fork before a replay or a capture.errdeferdestroyed the graph exec while the map kept the handle. The exec now stays alive in the map. A failed launch still returns the error, because the work may have started.tensorfold:graph_fallbacks_totalon/metrics(rank 0, dsv41 with graphs on), documented indocs/api.md.Every error raised under capture takes the fallback: a missing kernel variant, an instantiate error, and also a ring
error.FailedorNotPortedYet. The work reruns eagerly, thewarnline says a capture failed, and the counter goes up. An error that is not specific to capture returns from the eager run as before. The port tested in #45 returned pair-ending errors without an eager run; this PR does not separate those errors.Overlap with
fix/tp2-31-capture-join(a8ff91e, not inff234ecf4): that commit ends an aborted stretch capture with the forks joined and returns the body's error instead of 904. In this PR theLazyrecord path returns the body's error from the failed capture (the end-capture status is ignored), and the eager rerun's own result is what the caller sees afterwards. The PR also joins open forks at the start ofGraphs.run. Both change the capture block inGraphs.run. A trial merge of the fix into this branch gives one conflict, inround.zig, and none elsewhere. Whichever lands second needs a rebase. If the fix lands first, this PR can be rebased onto the fix, and the joins can move into a new abort hook on the site.Differences from #45, because of what this branch changed:
zig/src/core/lane_host_status.zig, so thegraph_fallbacksfield is read there.cuda.zigwas resolved by keeping the branch's lines and adding three lines (aHostfield, the assignment to the lane host, and the call to the family'sgraphFallbacks). The conflict innative.zigkept both sides' tests.Round.par,par_fork,par_join, used bydecodeGateUpParity). The objects are created once, and each event is recorded again before the wait on the event, inside one call. The stretch undo leaves the objects alone. This rests on reading the code: no test or run made a capture fail between the fork and the join there.warntext and the comments and test names were reworded: the line now reads "the key runs eagerly for the next N uses". The behavior is the same.If nothing fails, the only new cost is two flag checks before each replay.
Why
MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#65 is this bug on 2x DGX Spark:
In the three runs reported there, 36, 220 and 41 requests failed after that, and only a restart cleared the fault. MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks#63 shows the same 901 error at about 262,144 tokens, and #51 shows
cuGraphInstantiateWithFlagserror 800 followed by a server that takes requests but makes no tokens. The same root cause is a guess for both: neither report shows enough to be sure. The review of #45 also names bertholomus/deepseek-v4.1-tensorfold-tp2-2xgb10#31 as a possible beneficiary.Receipt
deepseek-v41-zig-tp2@ff234ecf4, Zig 0.17.0, x86_64 Linux, no GPU. The GPU environment of the earlier run is in fix(dsv41): run eagerly when a CUDA graph capture fails #45.27741e17(629c524aplus the kit's patches) with a port of this change. The port also restored the fork flags and ring state that the kit's KV-shard patches add, and returned the kit's pair-ending errors without an eager run, so the tested code differed from this PR.zig build test --summary all, with the worktree's.zig-cacheremoved first: 1777 passed, 10 skipped, 0 failed. The base head gives 1766 passed, 10 skipped.zig/tests/server/zig_test.sh: 403 passed, 2 skipped, 0 failed in the first suite (base 402), then 5 and 180 passed (base 5 and 179). The new server test checks the counter's line.graph_fallback.zigdriveLazythrough a fake of the graph calls (a clean capture; a failure in the work, at begin, at end capture or at instantiate; the hold cycle; the doubling; an eager error; a failed launch), plus one each inround.zig(the stretch undo),native.zig,lane_host_test.zigandmetrics.zig(the counter).left = 0inHolds.fail), the hold tests ingraph_fallback.zigfail ("expected 1024, found 0"). The change was reverted afterwards. The other mutation checks listed in fix(dsv41): run eagerly when a CUDA graph capture fails #45 were not repeated on this branch.zig build native -Doptimize=Debug: builds. This compiles the drafter's site andcuda.zig, which the host tests do not analyse.zig fmt --check build.zig zig: lists the same five files on the base head and on this branch (round.zigis one of the five, with or without this change).python3 tools/zig/lean_check.py: 1,863 problems on the base head and 1,863 on this branch.round.zig,draft.zig,model.zig,lane_host.zig,engine_api.zig,lane_host_test.zigandcuda.zigwere already over 600 lines and grew by 62, 8, 6, 1, 2, 21 and 3 lines.test-goldenand the Metal build are macOS only.Not covered:
_index_scorevariant or the instantiate error 800. A round that needs a kernel variant missing from the kit still fails the live streams, because the eager run fails too. The review thread on fix(dsv41): run eagerly when a CUDA graph capture fails #45 has a concrete case.Lazylogic only.Round.launches,launch_mark): the launches of a failed capture and of the eager rerun are both counted. The stretch undo does not restore the counters.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. Not re-run on this branch's head. The run in fix(dsv41): run eagerly when a CUDA graph capture fails #45 (drafts off) matched the stock run with drafts on, and the resume and solo checks passed.