Skip to content

Add a portable kernel for antialiased bicubic resize - #24447

Open
pytorchbot wants to merge 12 commits into
release/1.6from
cherry-pick-24400-by-pytorch_bot_bot_
Open

pytorchbot wants to merge 12 commits into
release/1.6from
cherry-pick-24400-by-pytorch_bot_bot_

Conversation

@pytorchbot

@pytorchbot pytorchbot commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Merge after #24298. This pick is stacked on it, because #24400 builds on changes that #24298 brings to release/1.6.

to_edge cannot lower F.interpolate(x, mode="bicubic", antialias=True) today. It exports to aten._upsample_bicubic2d_aa, which is not a core ATen op, so to_edge stops with:

SpecViolationError: Operator torch._ops.aten._upsample_bicubic2d_aa.default is not in Core ATen opset

ExecuTorch also has no kernel for it.

What this change does

It adds a portable kernel for _upsample_bicubic2d_aa.out and lists the op as an exception in the edge verifier, as the antialiased bilinear resize does.

Like ATen's CPU kernel, it uses the Keys cubic filter (a = -0.5), widened when shrinking, resizes the width, then the height, and copies a dim that keeps its size. float16 and bfloat16, which ATen's CPU kernel lacks, are computed in float32 and rounded once. uint8 follows ATen's fixed-point path: 16-bit integer weights, and each pass rounds and clamps. Other integer types fail in the kernel, as they do in ATen.

Between passes it keeps only the rows one output row reads, so shrinking 1080x1920 to 224x224 needs about 95 KB of temporary memory (43 KB for uint8). It rejects scales F.interpolate never produces: too small for the input, or (uint8 only) with weights too large for 16 bits. Where ATen's generic CPU path is wrong (contiguous input, output width 1, changing height; PyTorch issue 200568), it differs from eager on purpose.

The shared upsample helper now ignores a non-positive scale, as ATen does; of the other upsample kernels, only antialiased bilinear can pass one. A stale llava comment is updated, and the Arm all-ops test marks the kernel as needing a recipe. Like several portable kernels, it has no experimental CPU backend route yet.

Testing

  • New kernel tests check against ATen: shrinking, align_corners, scales, windows past the input, kept dims, both layouts, a batch, all float types, uint8 bit for bit, a dynamic output, running out of temporary memory, and error cases. Each case catches a bug that no other case catches.
  • The to_edge test compares static and dynamic programs (both layouts) with eager in float32 and, bit for bit, in uint8. It fails without the kernel or the verifier exception.
  • Outside the suite, a random sweep matched ATen's uint8 results exactly on x86 and Arm, on both its vectorized and generic paths; float32 within 7e-6 for unit-scale inputs, half types within one unit in the last place.
  • With XNNPACK, the resize stays outside the delegate. On one laptop thread, shrinking a contiguous 1080x1920 input to 224x224 takes about 7 ms (12 ms channels last).

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani

…and in exir (#24222)

Fixes found while exporting models to ExecuTorch backends.

## Fixes

- **exir:** `constant_prop_pass` keeps tensors that in-place ops write,
also through views or in `no_grad`/`autocast` blocks. `to_backend` keeps
the output node last. `to_edge` lowers `torch._check_tensor_all` and
`torch.sym_ite`. `GroupBasedPartitioner` no longer makes partitions that
depend on each other, unless caller node groups already do. Tensor
constants get static shapes; deduplicated constants keep their data.
- **AOTI (AOTInductor) and CUDA:** the missing-kernel check catches
scatter and index_put fallbacks. `detach` lowers, also inside
`torch.cond`. Strided outputs are copied. Transposed convolutions (1-D,
2-D, no dilation) and `torch._check_tensor_all` lower.
- **XNNPACK:** the quantizer skips bfloat16 tensors, and `cat` respects
scoped configs. ELU is delegated when XNNPACK can run it.
- **Qualcomm:** autocast regions are inlined correctly. Unquantized
bfloat16 nodes, data-dependent shapes and HTP partitions without runtime
inputs stay on the CPU. Fixes for stacks, 0-d selects, RMSNorm and int64
constants.
- **OpenVINO:** model data stays alive while OpenVINO 2026.4 reads it.
Partitions are converted with their input types.
- **Core ML:** bfloat16 ops, and ops above rank 5 outside static layout
chains, stay on the CPU.
- **MLX, Arm, Cortex-M, Vulkan, NXP, portable:** dtype checks for MLX
tensor lists, one constant slot per weight, slice updates on a constant
base; TOSA keeps unused delegate inputs for VGF; `MatmulToBmmPass` takes
a constant operand; Vulkan constants above 4 dims; non-strict programs
in the NXP quantizer; `cummin.out`.

## Behavior changes

- A CUDA `.pte` loads only the weights it was exported with. A `.ptd`
with other weights, for example after fine-tuning, needs a newly
exported `.pte`.
- CUDA graphs turn off when outputs do not match the ETensor layout. If
only the capture call sees such a layout, that call returns an error.
- `GroupBasedPartitioner` is about 2x slower on large graphs, and some
graphs get a different number of partitions. `to_backend` keeps the
original order of top-level nodes.
- XNNPACK: weight-only quantized linears, and 8da4w or 8da8w linears
(8-bit dynamic activations, 4-bit or 8-bit weights) lowered with
`to_backend`, fail at export instead of giving wrong results. Delegated
ELU flushes subnormal values to zero. Cats outside a scoped quantizer
config are no longer quantized.
- Ethos-U raises an error for a delegate with unused inputs instead of
returning wrong values.

We would like to cherry-pick this PR to release/1.6 after it merges. It
needs #23915 there first.

## Testing

Each fix has a test that fails without it. Run on an A100 (CUDA,
Vulkan), an x86 CPU (OpenVINO, QNN SDK), macOS (Core ML, MLX), the TOSA
reference model and VGF emulation (Arm), and the eIQ Neutron SDK (NXP).
Not run: Metal, ROCm, Windows CUDA.

(cherry picked from commit 054de1f)
This branch was picked before #24263 and #24295 reached release/1.6, so
three files carried their content only in part. Take release/1.6 and use
the versions a clean pick of 054de1f gives for
backends/qualcomm/partition/qnn_partitioner.py,
backends/qualcomm/tests/test_passes.py and exir/tests/test_pass_infra.py.
release/1.6 regenerated aoti_cuda_shims.lib to add
aoti_torch_dtype_element_size, and this branch regenerated it to add
aoti_torch_as_strided. Regenerate it per aoti_cuda_shims.lib.md with both
names, so it lists exactly the shims this tree exports.
The TensorRT guide said that when a second export overwrites aoti_cuda_blob.ptd, the first model can return wrong answers with no error. CUDA weight keys now carry a hash of their contents, so the first model's weights are missing from the new file and loading fails instead. Docs only.
…r HTP

On HTP, the partitioner keeps a partition that reads nothing at runtime on
the CPU, because HTP refuses such a graph. That happens after
decomposition, so an op that QNN asks to keep whole, such as linear or
hardswish, could land on the CPU still undecomposed, and
to_edge_transform_and_lower failed with "was not decomposed or delegated".

The filter returned with the ops to keep whole now also leaves out any op
that reads only constants, so it is decomposed like other CPU ops. An op
that reads a runtime input is still kept whole. The constant-only
partition test now has a linear in its constant branch, and fails without
this change.
…ly op

The comment said ops that read only constants are decomposed like other
CPU ops. Partitions are not known when the filter is built, so the rule
also decomposes such ops whose partition stays delegated. The comment and
the ops_to_not_decompose docstring now say so. Comment only.
… HTP

The HTP rule that decomposes ops reading only constants also covered the
quantize and dequantize ops in the skip table, such as
dequantize_per_channel_group on a frozen weight. QNN builds the weight
encoding from those ops. With today's torch they are not decomposed
anyway, so nothing changed in practice, but the filter should not ask for
it. They are now kept whole, and ops after them still count as reading
only constants.

A new test checks the filter itself and needs no QNN SDK, so it runs in
CI: a constant-only linear is decomposed, a linear on an input and a
constant-only group-wise dequantize are kept whole.
After lowering, exir asks the partitioner for its filter again and
reports any kept-whole op still on the CPU. On HTP the filter now let
constant-only ops through, so that report went quiet. For example
RecomposeRmsNorm rebuilt a constant-only rms_norm that then stayed on the
CPU, and the program failed later in to_executorch with "Missing out
variants" instead of the clear error from before. On the lowered (edge)
program the filter is plain filter_fn again, so these cases report as
they did before the HTP change. The filter test checks this too.
… Samsung passes

torch.export does not record some of the names it assigns: the names of
inputs, and the names of ops it renamed to make room for an input. With
an input called mul, the model's own input * 2 becomes mul_1, and
nothing marks mul_1 as taken. A later pass that creates a node can then
pick one of those names again, and the graph fails with "Node redefined
name t_1!". QNN models with autocast regions or attention hit it, and so
does the Samsung DecomposeRoll pass on a model with an input named cat.

reserve_node_names in backends/transforms/utils.py records the name of
every node of the top-level graph before the passes run. The QNN and
Samsung pass managers call it at the start of _transform. torch.fx has
no public way to reserve a name: copying the graph would register the
names, but it also renames inputs named after Python builtins (input
becomes input_1) while the graph signature keeps the old names. So the
helper adds the names to the graph's name set directly, with a TODO to
drop it once torch.export records the names
(pytorch/pytorch#202927).

The node name test runs the QNN export passes on an autocast model and
on an attention model whose inputs are named input and mul. It checks
the graph lints, keeps the input names of the graph signature, and
matches eager, and that the collision is real without the helper.

Same change as #24316 on main.
…on CPU

On HTP, a partition that reads nothing at runtime stays on the CPU,
because HTP refuses a graph without runtime inputs. That is decided after
decomposition. So a constant-only op that QNN keeps whole, such as
linear, could land in such a partition undecomposed, and lowering failed
with "was not decomposed or delegated".

The HTP filter of ops_to_not_decompose now runs the same drop rule before
decomposition and decomposes only the kept-whole ops it would leave on
the CPU. The real partitions are not known yet, so it partitions the
graph as if QNN took every op except the ones whose data types or shapes
rule it out (the checks filter_fn already makes, now shared as
must_stay_on_cpu). Ops in partitions that stay delegated keep their
whole form, and with it their quantization encodings.

The caller's skip_node_op_set and skip_node_id_set are left out of that
estimate on purpose. Decomposing adds nodes, and the new nodes take names
that the later nodes had before. If a skip could change what is
decomposed, adding one skip could rename the node that a
skip_node_id_set entry means, and the wrong node would be skipped. Ops
that QNN only rejects while partitioning are not known yet either. So a
kept-whole op that only a skip or such an op cuts off still fails as
before.

exir calls ops_to_not_decompose several times before decomposition and
uses only one of the filters, so the estimate runs on the filter's first
use. On a 10,240-node graph the partition search alone takes about 10 s.

On the lowered program the filter stays plain filter_fn, so exir still
reports kept-whole ops left on the CPU.

The constant-only partition test now stubs the QNN manager and the
compile, so it runs without the QNN SDK. It checks that a constant-only
linear cut off by bfloat16 casts is decomposed, that a constant-only
linear in a delegated partition stays whole, that the filter is plain
filter_fn on the lowered program, and that a node ID skip still names
the same add when an op skip and an ID-skipped feeder are present.

Same change as #24429 on main, including
its Partitioner.ops_to_not_decompose docstring. It replaces the earlier version
carried here, which decomposed every op that reads only constants.
`to_edge` cannot lower `F.interpolate(x, mode="bicubic",
antialias=True)` today. It exports to `aten._upsample_bicubic2d_aa`,
which is not a core ATen op, so `to_edge` stops with:

```
SpecViolationError: Operator torch._ops.aten._upsample_bicubic2d_aa.default is not in Core ATen opset
```

ExecuTorch also has no kernel for it.

## What this change does

It adds a portable kernel for `_upsample_bicubic2d_aa.out` and lists the
op as an exception in the edge verifier, as the antialiased bilinear
resize does.

Like ATen's CPU kernel, it uses the Keys cubic filter (a = -0.5),
widened when shrinking, resizes the width, then the height, and copies a
dim that keeps its size. float16 and bfloat16, which ATen's CPU kernel
lacks, are computed in float32 and rounded once. uint8 follows ATen's
fixed-point path: 16-bit integer weights, and each pass rounds and
clamps. Other integer types fail in the kernel, as they do in ATen.

Between passes it keeps only the rows one output row reads, so shrinking
1080x1920 to 224x224 needs about 95 KB of temporary memory (43 KB for
uint8). It rejects scales `F.interpolate` never produces: too small for
the input, or (uint8 only) with weights too large for 16 bits. Where
ATen's generic CPU path is wrong (contiguous input, output width 1,
changing height; [PyTorch issue
200568](pytorch/pytorch#200568)), it differs
from eager on purpose.

The shared upsample helper now ignores a non-positive scale, as ATen
does; of the other upsample kernels, only antialiased bilinear can pass
one. A stale llava comment is updated, and the Arm all-ops test marks
the kernel as needing a recipe. Like several portable kernels, it has no
experimental CPU backend route yet.

## Testing

- New kernel tests check against ATen: shrinking, `align_corners`,
scales, windows past the input, kept dims, both layouts, a batch, all
float types, uint8 bit for bit, a dynamic output, running out of
temporary memory, and error cases. Each case catches a bug that no other
case catches.
- The `to_edge` test compares static and dynamic programs (both layouts)
with eager in float32 and, bit for bit, in uint8. It fails without the
kernel or the verifier exception.
- Outside the suite, a random sweep matched ATen's uint8 results exactly
on x86 and Arm, on both its vectorized and generic paths; float32 within
7e-6 for unit-scale inputs, half types within one unit in the last
place.
- With XNNPACK, the resize stays outside the delegate. On one laptop
thread, shrinking a contiguous 1080x1920 input to 224x224 takes about 7
ms (12 ms channels last).

(cherry picked from commit a5476a5)
@pytorch-bot

pytorch-bot Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24447

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 Pending, 2 Unrelated Failures, 6 Unclassified Failures

As of commit bff92dc with merge base 4db778e (image):

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This branch was successfully deployed

1 active and 1 inactive deployments
upload-benchmark-results — bff92dc1 Deployed Nov 2, 2026 by JacobSzwejbka via upload-benchmark-results #21741
cadence — bff92dc1 Deployed Nov 2, 2026 by JacobSzwejbka via hifi-op-test / hifi4 #35993
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: arm Issues related to arm backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants