Track cuTile Rust v0.5.0 debugging candidates #1606

Track the v0.5.0 debugging fixes and the remaining scope of each issue. Merge reviewed CUDA Rust PRs into main, then include them in release/v0.5.x.

Release scope: every v0.5.0 feature must work with CTK 13.5 or earlier.

Updated 2026-10-30. Source PR acceptance and release inclusion are tracked separately.

All selected source fixes are merged into main, including #1452 (0925800d0) and #1684 (2d47e43e7). #1451 is closed. The only remaining v0.5.0 PR is draft #1613. Current candidate: d2c8120bf35ebb1fa1e6b9bb0b240f425a6fa91b — CI green: 76 passed, one expected deployment skip, whose complete file tree matches accepted main at 2d47e43e7437ca03cf84a469a9f54bfdcfb02c95 exactly. Hand SWQA this exact SHA for retesting, with CI status stated accurately. The actual release/v0.5.x branch remains at e027ac324 until #1613 is merged.

Nihal's host follow-ups are implemented: #1516 requires initialized source bytes for safe buffer casts; #1603 separates completion retention from user-visible buffer aliases while preserving access exclusion, allocation lifetime and graph ownership. #1604 carries both changes. The new-main example lockfile and archive-aware GPU test registrations are corrected. #1516 is merged as dfea11698 and #1506 is closed. Nihal approved the broader #1603/#1604 changes; merge readiness must use their latest heads and prerequisite order.

Local combined-host validation at 3ae634eee: all 1,427 CPU/assembler tests and 750 GPU tests pass (96 and three ignored respectively), plus 27 examples and eight benchmark smoke tests. PDL is the single expected example skip on CUDA 13.3. The first run hit disk exhaustion; the affected suites passed on rerun after clearing unused compiler caches. At integration revision 64ee86fe8, all 36 focused debug tests pass, including ignored assembler/GPU checks and live CUDA-GDB. The later change only migrates the expanded SIMT atomic example to owned conversions; strict example Clippy and all 24 atomic GPU cases pass on #1516 and final shared-stack head 95a51d708. cuTile/shared-host source is unchanged from the validated revisions. Fresh CI is running; do not equate local success with completed CI or acceptance.

SWQA's agreed pin remains a8d91829faa6a64e10bf27cf817c7e9f7608fba7. It remains in branch history. This refresh changes cast safety and tensor completion ownership, so the previous claim of no host/Tile source changes no longer applies. Explicitly hand off the new exact commit and retest the affected host, interop and lifetime workflows before sign-off. CTK 13.5 profiling/debugging validation remains outstanding.

The old launch-overhead hold remains removed. #1604 commit b811b7067, already integrated, reduced the reported individual-launch cost from 4.95 to 1.88 µs (earlier main baseline 1.68 µs); see the original benchmark report. These are historical measurements, not a fresh benchmark of this head.

Host-interop merge and issue closure tracking (2026-10-30):

All three host-interop source PRs are merged into main and included in the integration candidate. #1506, #1460, #1504, and #1461 are closed. Actual release-branch inclusion remains pending #1613.

The latest refresh resolves seven workflow-trigger conflicts by taking the accepted #1580 workflows from main. No additional host implementation changes were introduced. Broader work remains open in #1503, #1505, #1401, and #1403.

Host release inclusion (check only after acceptance and inclusion in release/v0.5.x):

• cuda-core: safe element casts require AnyBitPattern; bool is DeviceCopy #1516 safe casts and interoperable element types
• refactor(cuda-core): share host handles and allocations across Tile and SIMT #1603 shared host handles and allocations
• Unify the async host runtime and remove cuda_async::simt #1604 shared async runtime

Release inclusion checklist (check only after merge and inclusion in release/v0.5.x):

• fix: correct cuTile source locations for generated constants #1385 generated-constant fallback
• fix(cutile): attribute generated entry setup to the user signature #1480 entry-wrapper source locations
• fix(cutile-compiler): preserve origins of short-circuit, folded and bool-binding constants #1528 short-circuit/folded-expression/const-bool locations
• cutile: one CodegenMode for device debug, plus CUTILE_JIT_* run-time overrides #1600 debug-mode/runtime override controls
• fix(cutile): retain source origins for generated code and host checks #1602 arrays/reductions/host-check origins
• Use NVVM debug metadata merge behavior for native LTO #1564 NVVM debug metadata compatibility
• Preserve inline source scopes in line-table debug info #1582 line-info inline scopes
• fix(cutile): keep the slice offset in TensorView::view specialization #1541 offset-view specialization
• test(cutile): skip FP8 execution on unsupported targets #1683 FP8 test applicability
• fix(cutile): pass the smoke-test feature to the benchmark smoke run #1634 benchmark smoke feature
• fix(cutile): validate full tensor shapes before allocation #1452 tensor-shape validation (merged into main; cutile: unchecked shape products in api::full can overflow #1451 closed)
• fix(cuda-bindings): preserve complete struct layouts with libclang 22 #1684 libclang 22 binding layouts (merged into main; included in draft integration candidate)
• Complete release-branch CI and review of the combined candidates.
• After the accepted fixes are merged into main and included in release/v0.5.x, remind Melih to pass the prompt below to the internal model for CTK 13.5 profiling/debugging tests.

Remaining failures and uncovered scope:

• [BUG]: cuTile: at a cuda_tile_assert! trap, cuda-gdb reports the module's first line and drops the inlined helper frames #1379: actual assertion SIGTRAP and reopened GPU-coredump line/frame attribution still needs the Tile IR backend fix and live/core evidence. No backend fix PR is identified. The frontend fallback PR must not close that remaining scope.
• Validate helper-source/inline-frame behavior on CTK 13.5. The reproduced tileiras 13.4.59/13.4.92 loss is a documented backend limitation, with an exact-version warning in the assembler test. It is not fixed by fix: correct cuTile source locations for generated constants #1385, and its exception must not be broadened to 13.5 without investigation. Entry-location and frontend bytecode tests passing do not establish complete backend debugging fidelity.
• cuTile: audit generated source locations (optional follow-ups to #1379) #1475: close each sub-item only when its corresponding fix lands. Full-debug 13.3 reduction-line coalescing also remains a backend limitation; fix(cutile): retain source origins for generated code and host checks #1602 does not claim complete stepping accuracy.

Current refresh: the bindgen regression stays mandatory in the CPU/build suite and passes with libclang 22. The GPU archive selection excludes only that compiler-dependent test. In the CI CUDA 13.3.0 image without libclang, the original selection fails and the corrected selection passes all 16 runtime tests. The combined branch passes five #1452 shape unit tests and three GPU regressions. Fresh CI is requested.

Validation at 54be19d98: 17 binding tests and 50 core library tests pass after integrating #1684. The previous refresh aligned six SIMT example rustc-hash pins to 2.1.3; their locked resolutions and the exact example consistency gate pass. Shared host/Tile code and lockfiles are unchanged; CI has restarted. Source validation covers Tile checks, 108 SIMT host tests, codegen/frontend checks, all 263 changed lockfiles, and dependency policy checks. Before #1684, all 17 tensor-view tests, six float-operation tests, and eight benchmark smoke runs passed on RTX 5090 / CUDA 13.3. Formatting, shell syntax, CI image pins, and diff checks pass. #1634 is now integrated, so the benchmark runner no longer requires a temporary argument correction. Fresh CI is requested on the current integration head.

Deferred beyond v0.5.0:

#1476 / NVIDIA/cuda-gdb#29 is not a v0.5.0 release requirement or blocker. That PR changes NVIDIA's debugger so a breakpoint on an inlined helper call stops before the helper runs instead of sliding to the next statement. It is still open; the proposed fix has not been merged into a public CUDA Toolkit release. Per Melih's release-scope decision, remove #1476 from the v0.5.0 milestone and track it separately until a public toolkit contains the fix. Leave the issue open. The CUDA 13.4 tileiras regression above is a separate compiler issue.

Prompt to forward after merge and release-branch inclusion:

On the actual release/v0.5.x candidate, add regression tests and validate profiling/debugging with CTK 13.5. Record the commit, toolkit/debugger versions, GPU and driver. Cover cuTile Line/Full/None modes and runtime overrides; Rust source correlation in Nsight Compute; generated constants, entry wrappers, arrays/reductions and nested cross-file inline helpers; CUDA-GDB breakpoints/stepping and assertion/GPU-coredump line/frame attribution. Map results to the remaining sub-issues in #1379/#1475 and add CPU/serialized-IR, assembler or GPU tests as appropriate. Investigate failures rather than extending the 13.4.59/13.4.92 exception. Use the debugger shipped with CTK 13.5; do not require NVIDIA/cuda-gdb#29 or a later toolkit. Report what is supported, what remains broken, and minimal reproducers for backend failures.

Historical test-design follow-up: #1604 once exceeded a wall-clock drop threshold, then passed on an identical tree without relaxing the assertion. #1680 tracks wall-clock correctness assertions and sleep-based concurrency coordination across host, Tile, and SIMT tests. It remains separate follow-up work. #1516, #1603, and #1604 are all merged; use the current candidate SHA above for integration status.

Final acceptance refresh: candidate d2c8120bf records the accepted main ancestry and removes a duplicate changelog note; its entire file tree matches main. The previous integration head e16b73a9c failed the unchanged drop-timing assertion (200.853427 ms vs 100 ms), tracked under #1680. The exact test passed 30/30 isolated local runs on the final candidate; this does not diagnose the CI failure. No assertion was relaxed or skipped. Fresh CI has been requested.

Final CI result (2026-10-31 UTC): #1613 at d2c8120bf35ebb1fa1e6b9bb0b240f425a6fa91b passed all 76 applicable checks, including Rust CodeQL and the full GPU suite; the only skipped job is deployment. The previously failing drop-timing test passed unchanged in GPU CI. No additional code changes were needed. The exact SWQA candidate SHA is unchanged. #1613 remains draft pending SWQA retest/sign-off; actual release-branch inclusion and its reminder remain pending.