CUDA backend: delete each output handle once when a program returns one tensor twice by pytorchbot · Pull Request #24448 · pytorch/executorch

pytorchbot · 2026-11-02T17:11:11Z

Merge after #24324. This pick is stacked on it, because #24420 builds on changes that #24324 brings to release/1.6.

What is wrong today

When a CUDA program returns the same tensor more than once, for example y.detach() twice, AOTI (AOTInductor, PyTorch's ahead-of-time compiler that generates the CUDA code) puts one output handle in each of those output slots:

output_handles[0] = buf0.release();
output_handles[1] = output_handles[0];

The CUDA backend copied each output into its ExecuTorch tensor and then deleted that slot's handle. So it deleted the shared handle twice, and the process crashed, for example with:

SlimTensor storage_offset must not be negative, got -9007650542493611503
Failed to D2D copy GPU output 1 into ETensor
free(): double free detected in tcache 2

What this change does

A handle is now deleted only at the last output slot that holds it, as PyTorch's own AOTI runner does. The cleanup on error paths follows the same rule. With CUDA graphs on, a successful capture keeps every output handle for the graph and deletes none of them, so nothing changes there. If ending the capture or instantiating the graph fails, the cleanup deletes the handles by the same rule.

A comment in ReplaceViewCopyWithViewPass said that a view would make two outputs share one handle. That was misleading, because two alias_copy outputs of one tensor share one handle too, so the comment no longer says it.

What was tested

test_same_tensor_returned_twice returns one tensor in two output slots that are not next to each other, with CUDA graphs on: three eager calls, a graph capture and a replay, each compared with eager. With torch 2.14 on an A100 it passes, and it crashes without this change. The other backends/cuda tests that CI runs in the same job still pass.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani

…and in exir (#24222) Fixes found while exporting models to ExecuTorch backends. ## Fixes - **exir:** `constant_prop_pass` keeps tensors that in-place ops write, also through views or in `no_grad`/`autocast` blocks. `to_backend` keeps the output node last. `to_edge` lowers `torch._check_tensor_all` and `torch.sym_ite`. `GroupBasedPartitioner` no longer makes partitions that depend on each other, unless caller node groups already do. Tensor constants get static shapes; deduplicated constants keep their data. - **AOTI (AOTInductor) and CUDA:** the missing-kernel check catches scatter and index_put fallbacks. `detach` lowers, also inside `torch.cond`. Strided outputs are copied. Transposed convolutions (1-D, 2-D, no dilation) and `torch._check_tensor_all` lower. - **XNNPACK:** the quantizer skips bfloat16 tensors, and `cat` respects scoped configs. ELU is delegated when XNNPACK can run it. - **Qualcomm:** autocast regions are inlined correctly. Unquantized bfloat16 nodes, data-dependent shapes and HTP partitions without runtime inputs stay on the CPU. Fixes for stacks, 0-d selects, RMSNorm and int64 constants. - **OpenVINO:** model data stays alive while OpenVINO 2026.4 reads it. Partitions are converted with their input types. - **Core ML:** bfloat16 ops, and ops above rank 5 outside static layout chains, stay on the CPU. - **MLX, Arm, Cortex-M, Vulkan, NXP, portable:** dtype checks for MLX tensor lists, one constant slot per weight, slice updates on a constant base; TOSA keeps unused delegate inputs for VGF; `MatmulToBmmPass` takes a constant operand; Vulkan constants above 4 dims; non-strict programs in the NXP quantizer; `cummin.out`. ## Behavior changes - A CUDA `.pte` loads only the weights it was exported with. A `.ptd` with other weights, for example after fine-tuning, needs a newly exported `.pte`. - CUDA graphs turn off when outputs do not match the ETensor layout. If only the capture call sees such a layout, that call returns an error. - `GroupBasedPartitioner` is about 2x slower on large graphs, and some graphs get a different number of partitions. `to_backend` keeps the original order of top-level nodes. - XNNPACK: weight-only quantized linears, and 8da4w or 8da8w linears (8-bit dynamic activations, 4-bit or 8-bit weights) lowered with `to_backend`, fail at export instead of giving wrong results. Delegated ELU flushes subnormal values to zero. Cats outside a scoped quantizer config are no longer quantized. - Ethos-U raises an error for a delegate with unused inputs instead of returning wrong values. We would like to cherry-pick this PR to release/1.6 after it merges. It needs #23915 there first. ## Testing Each fix has a test that fails without it. Run on an A100 (CUDA, Vulkan), an x86 CPU (OpenVINO, QNN SDK), macOS (Core ML, MLX), the TOSA reference model and VGF emulation (Arm), and the eIQ Neutron SDK (NXP). Not run: Metal, ROCm, Windows CUDA. (cherry picked from commit 054de1f)

This branch was picked before #24263 and #24295 reached release/1.6, so three files carried their content only in part. Take release/1.6 and use the versions a clean pick of 054de1f gives for backends/qualcomm/partition/qnn_partitioner.py, backends/qualcomm/tests/test_passes.py and exir/tests/test_pass_infra.py.

…fer check (#24299) ## What is wrong today Robot policies and follow-up checks on #24222 found these bugs in the CUDA backend and the runtime: 1. **CUDA mask indexing:** `x[mask]` and `x[mask] = values` have no C shims, so lowering fails with "missing fallback kernels". This blocks SmolVLA. 2. **CUDA graphs:** a channels-last output turns CUDA graphs off, although a raw copy is correct. 3. **CUDA graph shapes:** when the captured call has a new batch size, it and every replay return the previous output shape. 4. **Output buffers:** `Method::set_output_data_ptr` checks a buffer against the output's current size. After a dynamic output shrinks, a too-small buffer is accepted and a later, larger call writes past it. ## What this change does One commit per fix: 1. New shims for index and index_put follow ATen's indexing rules. The Windows import library gains the new names. Inside a CUDA graph capture, a mask index fails with a clear error. 2. The check compares output strides with the strides the ETensor's dim order gives. 3. The capture step resizes each output to the captured shape. 4. The buffer is checked against the output's upper bound. Code that binds a smaller buffer after a call now gets `InvalidArgument`. ## Tests Each new test fails without its fix. - **CUDA (A100):** mask reads and writes, also inside a capture; the raw-copy check; a capture call with a new batch size. - **Runtime:** a too-small output buffer is refused after a dynamic output shrinks; the runtime tests pass. On LeRobot policies, SmolVLA now lowers to CUDA and runs in 43 ms per call on an A100, against 504 ms in eager PyTorch (cosine similarity 0.99996). (cherry picked from commit ff1bab3)

…fer check (#24299) ## What is wrong today Robot policies and follow-up checks on #24222 found these bugs in the CUDA backend and the runtime: 1. **CUDA mask indexing:** `x[mask]` and `x[mask] = values` have no C shims, so lowering fails with "missing fallback kernels". This blocks SmolVLA. 2. **CUDA graphs:** a channels-last output turns CUDA graphs off, although a raw copy is correct. 3. **CUDA graph shapes:** when the captured call has a new batch size, it and every replay return the previous output shape. 4. **Output buffers:** `Method::set_output_data_ptr` checks a buffer against the output's current size. After a dynamic output shrinks, a too-small buffer is accepted and a later, larger call writes past it. ## What this change does One commit per fix: 1. New shims for index and index_put follow ATen's indexing rules. The Windows import library gains the new names. Inside a CUDA graph capture, a mask index fails with a clear error. 2. The check compares output strides with the strides the ETensor's dim order gives. 3. The capture step resizes each output to the captured shape. 4. The buffer is checked against the output's upper bound. Code that binds a smaller buffer after a call now gets `InvalidArgument`. ## Tests Each new test fails without its fix. - **CUDA (A100):** mask reads and writes, also inside a capture; the raw-copy check; a capture call with a new batch size. - **Runtime:** a too-small output buffer is refused after a dynamic output shrinks; the runtime tests pass. On LeRobot policies, SmolVLA now lowers to CUDA and runs in 43 ms per call on an A100, against 504 ms in eager PyTorch (cosine similarity 0.99996).

The pick was made before #24298 (the #24222 pick) was on release/1.6, so three files carried partial copies of #24222. This merges #24298's branch and restores the files so that, on top of #24298, the change is exactly #24299 (ff1bab3).

…24314) This change fixes several problems in the CUDA backend. Each fix is its own commit. ## Name the weights file of each program The backend saves every program's weights to `aoti_cuda_blob.ptd`. Exporting several programs into one folder overwrites the weights, and the earlier programs then fail to load. Renaming each file after export works, but is easy to miss. A new option names the file: ```python partitioner = CudaPartitioner([ CudaBackend.generate_method_name_compile_spec("forward"), CudaBackend.generate_data_name_compile_spec("encoder"), ]) # writes encoder.ptd ``` Without the option nothing changes. The name is checked: up to 200 ASCII letters, digits, `_`, `-` and `.`, not starting with `.` or `-`, not ending with `.ptd`, and not a Windows device name such as `NUL`. All methods of a program must share it. A backend that keeps its data inside the `.pte`, such as Metal, refuses it. ## Lower antialiased resizes `F.interpolate(x, mode="bilinear" or "bicubic", antialias=True)` failed to lower with "missing fallback kernels". The backend now rewrites it into plain tensor math, with PyTorch's window and weight formulas, for dynamic sizes and float32, float16 and bfloat16. The results are not bit-identical: in float32 they stay within about 2e-5 of PyTorch's kernel for inputs between 0 and 1. ## Tests Ran on an A100 GPU (Linux x86_64): `test_cuda_export.py`, `test_antialiased_resize.py`, `test_missing_weights_blob.py`, `test_merge_ptes.py` and `test_cuda_weight_metadata.py`. Each test added for a fix fails when that fix is removed. (cherry picked from commit 7586c16)

…24314) This change fixes several problems in the CUDA backend. Each fix is its own commit. ## Name the weights file of each program The backend saves every program's weights to `aoti_cuda_blob.ptd`. Exporting several programs into one folder overwrites the weights, and the earlier programs then fail to load. Renaming each file after export works, but is easy to miss. A new option names the file: ```python partitioner = CudaPartitioner([ CudaBackend.generate_method_name_compile_spec("forward"), CudaBackend.generate_data_name_compile_spec("encoder"), ]) # writes encoder.ptd ``` Without the option nothing changes. The name is checked: up to 200 ASCII letters, digits, `_`, `-` and `.`, not starting with `.` or `-`, not ending with `.ptd`, and not a Windows device name such as `NUL`. All methods of a program must share it. A backend that keeps its data inside the `.pte`, such as Metal, refuses it. ## Lower antialiased resizes `F.interpolate(x, mode="bilinear" or "bicubic", antialias=True)` failed to lower with "missing fallback kernels". The backend now rewrites it into plain tensor math, with PyTorch's window and weight formulas, for dynamic sizes and float32, float16 and bfloat16. The results are not bit-identical: in float32 they stay within about 2e-5 of PyTorch's kernel for inputs between 0 and 1. ## Tests Ran on an A100 GPU (Linux x86_64): `test_cuda_export.py`, `test_antialiased_resize.py`, `test_missing_weights_blob.py`, `test_merge_ptes.py` and `test_cuda_weight_metadata.py`. Each test added for a fix fails when that fix is removed.

The pick was made before #24298 and #24320 were on release/1.6, so it carried partial copies of earlier changes. This stacks it on #24320's branch so that, on top of #24320, the change is exactly #24314 (7586c16).

A CUDA graph replays the shapes it captured. Before each replay, the backend copied the new input into the captured input buffer, and it checked only that the byte count matched. An input can change shape without changing its byte count, for example from (2, 3, 4, 4) to (2, 4, 3, 4). Such a call replayed the captured shape and returned an output with the old shape and wrong values, with no error. The backend now records each input's shape at capture, and a replay with a different shape fails with InvalidArgument, the same error a different byte count already gave. test_output_stride_rearrange.py captures with (2, 3, 4, 4) inputs, then calls with (2, 4, 3, 4), for contiguous and channels-last outputs. Without this change those calls return the wrong shape.

The wheel check names every kernel shim that libaoti_cuda_shims must define, so a library built without some of its CUDA sources fails the check. The mask index and index_put shims were missing from that list.

The test gave Inductor only the CUDA backend's decomposition table, which replaced Inductor's own table. With torch 2.14 and CI set, Inductor then rejects the fallback for aten.arange and the test fails. Real lowering applies the backend's table at export and keeps Inductor's defaults, so the test now merges the two.

test_align_corners_compiles_with_dynamic_sizes could not fail. It passed with no rewrite at all, since torch.compile then calls ATen's kernel, and it passed with the min and max in the align_corners ratio replaced by a plain if, since torch.compile treats an output size of 1 as a constant and compiles again. With torch 2.14 and CI set it also failed for a reason unrelated to the resize. test_bicubic_antialias_resize already runs an exported program at an output height of 1, which is where a plain if breaks.

The comment on the min and max in the align_corners ratio blamed torch.compile, which handles a conditional by compiling again. The real reason is export: an exported program keeps the branch it was traced with, so a plain if divides by zero at an output size of 1 when the program was traced with a larger one.

….pte Only generate_data_name_compile_spec refused a data name for a backend that keeps its data in the .pte, such as Metal. A data_name compile spec built by hand went through preprocess with no error and no name check, even for a name like "../../outside". data_name_from_compile_specs now refuses it too, and preprocess always calls it, so both paths behave the same.

Windows and macOS usually treat Encoder.ptd and encoder.ptd as one file, so two programs exported into one folder with those data names overwrite each other's weights, and the first one then fails to load. The name check allows both, as both are valid names elsewhere, so the docs now say to pick names that differ in more than case.

Some comments added with the CUDA mask indexing change say the wrong thing: - index.h described accumulation as out[indices] += values. With a repeated index, += keeps only one of the values, while index_put_ with accumulate adds every value, and so does the shim. - index.cu said CUB counts items in an int for the index_put sort. That is true for the mask path, but the radix sort takes a wider count. The limit there comes from the int the code passes to the sort. - index.cu said the accumulation types are ATen's. That holds for half and bfloat16, which add in float. Narrower integers add in int64 here, which gives the same wrapped result. - method.h named the output index i in the set_output_data_ptr docs, but the parameter is output_idx. Only comments change.

A backend that keeps its data in the .pte, such as Metal, refuses a data_name compile spec. preprocess only checked it after moving the program to the device and compiling it with AOTInductor, which took about 18 seconds and left the compiled library on disk. It now checks the name first. The data name test calls the shared preprocess with such a spec, so moving the check back behind the compile, or dropping it, fails the test.

…ick reverted The bot pick took the older main versions of these two files, which undid the release/1.6 fix from #24289.

Two release/1.6 changes each regenerated this import library from the same base. This one adds the CUDA index shims, aoti_torch_as_strided and aoti_torch_get_size. The other adds aoti_torch_dtype_element_size. Git cannot merge a binary file, so whichever lands second would conflict, and keeping either side alone drops names the other needs. List the union of both sets, regenerated per aoti_cuda_shims.lib.md. The archive has no member timestamps, and rebuilding it gives the same bytes.

release/1.6 regenerated aoti_cuda_shims.lib to add aoti_torch_dtype_element_size, and this branch regenerated it to add aoti_torch_as_strided. Regenerate it per aoti_cuda_shims.lib.md with both names, so it lists exactly the shims this tree exports.

Regenerate aoti_cuda_shims.lib per aoti_cuda_shims.lib.md so it lists exactly the shims this tree exports: release/1.6's set, including aoti_torch_dtype_element_size, plus aoti_torch_as_strided, aoti_torch_cuda_index_Tensor, aoti_torch_get_size and aoti_torch_index_put_out.

…r HTP On HTP, the partitioner keeps a partition that reads nothing at runtime on the CPU, because HTP refuses such a graph. That happens after decomposition, so an op that QNN asks to keep whole, such as linear or hardswish, could land on the CPU still undecomposed, and to_edge_transform_and_lower failed with "was not decomposed or delegated". The filter returned with the ops to keep whole now also leaves out any op that reads only constants, so it is decomposed like other CPU ops. An op that reads a runtime input is still kept whole. The constant-only partition test now has a linear in its constant branch, and fails without this change.

…ly op The comment said ops that read only constants are decomposed like other CPU ops. Partitions are not known when the filter is built, so the rule also decomposes such ops whose partition stays delegated. The comment and the ops_to_not_decompose docstring now say so. Comment only.

… HTP The HTP rule that decomposes ops reading only constants also covered the quantize and dequantize ops in the skip table, such as dequantize_per_channel_group on a frozen weight. QNN builds the weight encoding from those ops. With today's torch they are not decomposed anyway, so nothing changed in practice, but the filter should not ask for it. They are now kept whole, and ops after them still count as reading only constants. A new test checks the filter itself and needs no QNN SDK, so it runs in CI: a constant-only linear is decomposed, a linear on an input and a constant-only group-wise dequantize are kept whole.

shoumikhin added 30 commits October 29, 2026 16:09

Rebase the #24314 cherry-pick onto the #24299 cherry-pick

44980c5

The pick was made before #24298 and #24320 were on release/1.6, so it carried partial copies of earlier changes. This stacks it on #24320's branch so that, on top of #24320, the change is exactly #24314 (7586c16).

List the CUDA index shims in the wheel's shared library check

d7322be

The wheel check names every kernel shim that libaoti_cuda_shims must define, so a library built without some of its CUDA sources fails the check. The mask index and index_put shims were missing from that list.

Merge the updated #24222 cherry-pick into the #24299 cherry-pick

a95adaf

Merge release/1.6 into the #24222 cherry-pick

d22b428

Merge the updated #24299 cherry-pick into the #24314 cherry-pick

afedcbd

Merge the updated #24299 cherry-pick into the #24314 cherry-pick

2ee2baa

Merge the updated #24299 cherry-pick into the #24314 cherry-pick

b535a66

Restore release/1.6 Windows Voxtral CI files that the #24299 cherry-p…

cdb488e

…ick reverted The bot pick took the older main versions of these two files, which undid the release/1.6 fix from #24289.

Merge the updated #24299 cherry-pick into the #24314 cherry-pick

530c96b

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 2, 2026

This was referenced Nov 2, 2026

[v1.6.0] Release Schedule and Tracker #23608

Open

CUDA backend: delete each output handle once when a program returns one tensor twice #24442

Closed

pytorchbot temporarily deployed to cadence November 2, 2026 17:45 — with GitHub Actions Inactive

pytorchbot temporarily deployed to cadence November 2, 2026 17:50 — with GitHub Actions Inactive

pytorchbot temporarily deployed to cadence November 2, 2026 17:51 — with GitHub Actions Inactive

shoumikhin changed the base branch from cherry-pick-24314-by-pytorch_bot_bot_ to release/1.6 November 2, 2026 17:58

shoumikhin requested review from JacobSzwejbka, SS-JIA, digantdesai, larryliu0820, mergennachin, metascroy, psiddh, rascani, robert-kalmar and shoumikhin as code owners November 2, 2026 17:58

JacobSzwejbka closed this Nov 2, 2026

JacobSzwejbka reopened this Nov 2, 2026

github-actions Bot added ciflow/trunk module: arm Issues related to arm backend labels Nov 2, 2026

JacobSzwejbka temporarily deployed to cadence November 2, 2026 18:28 — with GitHub Actions Inactive

JacobSzwejbka deployed to cadence November 2, 2026 18:45 — with GitHub Actions Active

JacobSzwejbka deployed to cadence November 2, 2026 18:47 — with GitHub Actions Active

JacobSzwejbka deployed to cadence November 2, 2026 18:48 — with GitHub Actions Active

shoumikhin mentioned this pull request Nov 2, 2026

CUDA backend: lower fused attention variants #24455

Open

pytorchbot deployed to upload-benchmark-results November 2, 2026 19:37 — with GitHub Actions Active

Navigation Menu

Repository navigation

CUDA backend: delete each output handle once when a program returns one tensor twice - #24448

CUDA backend: delete each output handle once when a program returns one tensor twice#24448
pytorchbot wants to merge 32 commits into
release/1.6from
cherry-pick-24420-by-pytorch_bot_bot_

pytorchbot commented Nov 2, 2026 •

edited by pytorch-bot Bot

Loading

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pytorchbot commented Nov 2, 2026 • edited by pytorch-bot Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is wrong today

What this change does

What was tested

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pytorchbot commented Nov 2, 2026 •

edited by pytorch-bot Bot

Loading