OpenVINO: fix bool min/max and masked read lowering, and follow-ups to #24294 by pytorchbot · Pull Request #24446 · pytorch/executorch

pytorchbot · 2026-11-02T17:01:41Z

Merge after #24377. This pick is stacked on it, because #24396 builds on #24294.

What is wrong today

The OpenVINO partitioner gives min, max, amin, amax, minimum, maximum, clamp and sign to OpenVINO even when every tensor they read is bool. OpenVINO cannot convert these for bool:

ReduceMin ... Element type of data input must be numeric. Got: boolean

torch.min(x > 0, dim=1) and MultivariateNormal's argument check (PyTorch 2.15) hit this. Reading masked values after writing them fails too:

x[mask] = x[mask] * 2
return x + x[mask].sum()  # Dynamic rank for aten::index input is not supported.

What this change does

The partitioner leaves these ops to ExecuTorch's CPU kernels when every tensor they read is bool. Ops that also read a float tensor, like maximum(mask, x), stay with OpenVINO. Casting to uint8 could give wrong min(dim) and max(dim) indices on ties, as the docs now note.

For the masked read, preprocess fills each node's tensor_meta from val. OpenVINO reads ranks only from tensor_meta, which ExecuTorch leaves empty.

Follow-ups to #24294:

The test runner exits with 1 on failure (from Fix OpenVINO tests in CI #24305), and CI runs model tests even after an op test fails.
OpenVINO dropped a first input that only a folded cumsum read, failing every call. Preprocess puts it back and checks the input count.
An output planned as dynamic but compiled as fixed failed every call. Its buffer now starts with the compiled shape.
The cumsum fold works under a CUDA default device on a CPU-only PyTorch (exir's ScalarToTensorPass still fails there).
An empty last inference_precision entry now also means the default at load.
The docs say which CPUs compute some integer ops in float32, and where the float32 default is slower.

What was tested

Linux x86_64, OpenVINO 2026.4.1 and 2025.1.0, CPU, unless noted:

Op and model tests pass. The bool min model (both releases) and the masked read (2026.4.1) failed to lower, and now match eager.
On 2026.4.1, each backend fix has a test that fails without it. The runner and CI script were checked by hand.
Partitions are unchanged for every probed op with integer, float16, bfloat16 or float32 inputs, and for MobileNetV2, ResNet-18 and a transformer layer.
With the tensor_meta change, all other test partitions convert to the same OpenVINO graphs.
The docs' integer claims were measured on x86 through the delegate, and on ARM with plain OpenVINO 2026.4.1.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani

…and in exir (#24222) Fixes found while exporting models to ExecuTorch backends. ## Fixes - **exir:** `constant_prop_pass` keeps tensors that in-place ops write, also through views or in `no_grad`/`autocast` blocks. `to_backend` keeps the output node last. `to_edge` lowers `torch._check_tensor_all` and `torch.sym_ite`. `GroupBasedPartitioner` no longer makes partitions that depend on each other, unless caller node groups already do. Tensor constants get static shapes; deduplicated constants keep their data. - **AOTI (AOTInductor) and CUDA:** the missing-kernel check catches scatter and index_put fallbacks. `detach` lowers, also inside `torch.cond`. Strided outputs are copied. Transposed convolutions (1-D, 2-D, no dilation) and `torch._check_tensor_all` lower. - **XNNPACK:** the quantizer skips bfloat16 tensors, and `cat` respects scoped configs. ELU is delegated when XNNPACK can run it. - **Qualcomm:** autocast regions are inlined correctly. Unquantized bfloat16 nodes, data-dependent shapes and HTP partitions without runtime inputs stay on the CPU. Fixes for stacks, 0-d selects, RMSNorm and int64 constants. - **OpenVINO:** model data stays alive while OpenVINO 2026.4 reads it. Partitions are converted with their input types. - **Core ML:** bfloat16 ops, and ops above rank 5 outside static layout chains, stay on the CPU. - **MLX, Arm, Cortex-M, Vulkan, NXP, portable:** dtype checks for MLX tensor lists, one constant slot per weight, slice updates on a constant base; TOSA keeps unused delegate inputs for VGF; `MatmulToBmmPass` takes a constant operand; Vulkan constants above 4 dims; non-strict programs in the NXP quantizer; `cummin.out`. ## Behavior changes - A CUDA `.pte` loads only the weights it was exported with. A `.ptd` with other weights, for example after fine-tuning, needs a newly exported `.pte`. - CUDA graphs turn off when outputs do not match the ETensor layout. If only the capture call sees such a layout, that call returns an error. - `GroupBasedPartitioner` is about 2x slower on large graphs, and some graphs get a different number of partitions. `to_backend` keeps the original order of top-level nodes. - XNNPACK: weight-only quantized linears, and 8da4w or 8da8w linears (8-bit dynamic activations, 4-bit or 8-bit weights) lowered with `to_backend`, fail at export instead of giving wrong results. Delegated ELU flushes subnormal values to zero. Cats outside a scoped quantizer config are no longer quantized. - Ethos-U raises an error for a delegate with unused inputs instead of returning wrong values. We would like to cherry-pick this PR to release/1.6 after it merges. It needs #23915 there first. ## Testing Each fix has a test that fails without it. Run on an A100 (CUDA, Vulkan), an x86 CPU (OpenVINO, QNN SDK), macOS (Core ML, MLX), the TOSA reference model and VGF emulation (Arm), and the eIQ Neutron SDK (NXP). Not run: Metal, ROCm, Windows CUDA. (cherry picked from commit 054de1f)

This branch was picked before #24263 and #24295 reached release/1.6, so three files carried their content only in part. Take release/1.6 and use the versions a clean pick of 054de1f gives for backends/qualcomm/partition/qnn_partitioner.py, backends/qualcomm/tests/test_passes.py and exir/tests/test_pass_infra.py.

release/1.6 regenerated aoti_cuda_shims.lib to add aoti_torch_dtype_element_size, and this branch regenerated it to add aoti_torch_as_strided. Regenerate it per aoti_cuda_shims.lib.md with both names, so it lists exactly the shims this tree exports.

…ault to float32 on CPU, in the OpenVINO backend (#24294) ## What is wrong today The OpenVINO backend has these problems: 1. A 0-d (scalar) input or output fails on every run, because OpenVINO's `ov_shape_create` refuses rank 0. 2. A float32 program does not compute in float32 on many CPUs. OpenVINO picks bfloat16 (two to three significant digits) on x86 CPUs with bfloat16 hardware and float16 on ARM CPUs, and no compile spec can change it. 3. An output whose size depends on an int input keeps its largest size. `x[:n].view(n, -1)` with x of shape (4, 8) and n = 3 returns (4, 8). 4. `cumsum` and `prod` of a bool or integer tensor are not computed in int64, as PyTorch does. A cumsum over a bool mask wraps after 255, so pi0 and pi0.5 get wrong position ids and wrong actions. ## What this change does - **0-d tensors:** a rank-0 tensor keeps its empty shape, which OpenVINO accepts. - **Precision:** when the device is `CPU`, a program computes in float32 by default, both when it is lowered and when it is loaded. A program can ask for a faster precision: ```python OpenvinoPartitioner( [CompileSpec("device", b"CPU"), CompileSpec("inference_precision", b"bf16")] ) ``` The value is `f32`, `f16` or `bf16`, checked before conversion starts. Other devices keep OpenVINO's default. On x86 CPUs OpenVINO applies the precision when the runtime loads the program, so the runtime passes it on too. - **Dynamic output sizes:** an output whose size can change goes to a buffer that OpenVINO owns, reused by every call. It starts empty and OpenVINO grows it to the largest size a call writes, so a large upper bound costs nothing until a call needs it. After inference the backend resizes the ExecuTorch output to OpenVINO's shape and copies the data in, so one delegate can also serve several calls, as programs from ExecuTorch 0.7 and earlier do. Static outputs still use the ExecuTorch buffer directly. - **Integer reductions:** the input of `cumsum` and `prod` is cast to int64 before conversion. An integer cumsum that reads only constants is computed during lowering, because OpenVINO's CPU plugin gives it wrong results from the second inference on. This includes a cumsum over `ones_like` or `zeros_like` of an input with a static shape, which OpenVINO folds into a constant too. Each one is folded on its own, and one that cannot be folded is named in a warning. Trade-offs: - float32 is slower. On a Xeon with bfloat16 hardware, a 4-layer MLP takes 2.4 ms in float32 and 1.5 ms in bfloat16. On x86 this also applies to programs exported before this change, because OpenVINO applies the precision when a program loads. To get the old speed, export again with `inference_precision` set to `bf16`. - OpenVINO's CPU plugin computes int64 in 32 bits, so an integer result past the int32 range wraps without an error. That was already true for int64 programs, and now also applies to `cumsum` and `prod` over smaller integer types. ## Tests Each new test fails without its fix: 0-d inputs and outputs; a 256 x 256 addmm checked to 1e-4, and the same addmm lowered with `bf16` checked not to reach 1e-4, so the runtime must apply an explicit precision (both skipped where OpenVINO already computes in float32); an unknown precision refused before conversion; dynamic output sizes, on one program and on one delegate shared by two calls; bool and int32 cumsum and prod; pi0-style masks with int64 and int32 cumsums over two calls, built from a constant and from `ones_like`; and folding when one cumsum cannot be evaluated. Two guards pass either way: integer `sum`, which OpenVINO already computes in int64, and a static output after a dynamic view. The full OpenVINO op suite passes on Linux x86_64 (a CPU with bfloat16 units) with OpenVINO 2026.4.1, through both pytest and the CI test runner. With OpenVINO 2025.1.0 it also passes, except that the prod test is skipped because that release does not delegate prod. (cherry picked from commit 7b14148)

The pick was made on release/1.6, which does not have #24298 (the #24222 pick) yet. #24294 builds on #24222, so the pick depends on code that release/1.6 lacks, such as _nodes_written_in_place in exir's constant_prop_pass, and its conflict resolution brought in some of #24222's OpenVINO changes but not others. This merges a clean pick of 7b14148 onto #24298's branch and takes the clean pick's side in the OpenVINO files, so that, on top of #24298, the change is exactly #24294 (7b14148). A plain merge of the two would keep two copies of _PRECISIONS in preprocess.py.

The OpenVINO CI job runs backends/openvino/tests/test_runner.py. When a test failed, the runner printed "completed with failures" but still exited with code 0, so the CI job stayed green and the failure went unnoticed. Now the runner exits with code 1 when any test fails, so CI turns red. This is the test runner fix from #24305. Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>

The TensorRT guide said that when a second export overwrites aoti_cuda_blob.ptd, the first model can return wrong answers with no error. CUDA weight keys now carry a hash of their contents, so the first model's weights are missing from the new file and loading fails instead. Docs only.

…r HTP On HTP, the partitioner keeps a partition that reads nothing at runtime on the CPU, because HTP refuses such a graph. That happens after decomposition, so an op that QNN asks to keep whole, such as linear or hardswish, could land on the CPU still undecomposed, and to_edge_transform_and_lower failed with "was not decomposed or delegated". The filter returned with the ops to keep whole now also leaves out any op that reads only constants, so it is decomposed like other CPU ops. An op that reads a runtime input is still kept whole. The constant-only partition test now has a linear in its constant branch, and fails without this change.

…ly op The comment said ops that read only constants are decomposed like other CPU ops. Partitions are not known when the filter is built, so the rule also decomposes such ops whose partition stays delegated. The comment and the ops_to_not_decompose docstring now say so. Comment only.

… HTP The HTP rule that decomposes ops reading only constants also covered the quantize and dequantize ops in the skip table, such as dequantize_per_channel_group on a frozen weight. QNN builds the weight encoding from those ops. With today's torch they are not decomposed anyway, so nothing changed in practice, but the filter should not ask for it. They are now kept whole, and ops after them still count as reading only constants. A new test checks the filter itself and needs no QNN SDK, so it runs in CI: a constant-only linear is decomposed, a linear on an input and a constant-only group-wise dequantize are kept whole.

After lowering, exir asks the partitioner for its filter again and reports any kept-whole op still on the CPU. On HTP the filter now let constant-only ops through, so that report went quiet. For example RecomposeRmsNorm rebuilt a constant-only rms_norm that then stayed on the CPU, and the program failed later in to_executorch with "Missing out variants" instead of the clear error from before. On the lowered (edge) program the filter is plain filter_fn again, so these cases report as they did before the HTP change. The filter test checks this too.

This pick is stacked on #24298 (the #24222 pick), which has since gained follow-up commits for the Qualcomm partitioner and the TensorRT guide. Taking them keeps this branch equal to #24298 plus #24294, so that refreshing it from release/1.6 after #24298 merges cannot undo them. The merge is clean, and the follow-ups touch no OpenVINO files.

…d it A position cumsum such as cumsum(ones_like(input_ids)) - 1 reads only the shape of input_ids. Once FoldConstantCumsumPass folds it, nothing in the partition reads input_ids. When that input comes first, OpenVINO's PyTorch converter takes it for forward's self and removes it. The runtime still passes every partition input, so every call failed with "expected 2 args (inputs=1 + outputs=1), got 3". An unused input in any other position is kept by the converter. Not folding is no fix, because masks built from such a cumsum then go wrong on OpenVINO's CPU plugin from the second call on. Instead, when the converted model has exactly one parameter fewer than the partition has inputs, preprocess puts the first one back, so the compiled model takes the same inputs the delegate passes. The test leaves both the first input and a later one unused.

ExecuTorch can plan an output with a dynamic size while OpenVINO compiles a fixed shape for it, for example x[torch.ones(4, dtype=bool)] once OpenVINO folds the mask. The buffer the runtime makes for such an output started empty, and OpenVINO refuses an output tensor whose shape differs from a fixed one, so every call failed with 0x1. Before the owned buffers, these programs ran. The buffer now starts with the shape of the infer request's own output tensor: empty along each dynamic dimension and the full shape where OpenVINO fixed it.

FoldConstantCumsumPass evaluates a full_like from an empty tensor of its input's shape. That tensor went to the default device, so lowering for OpenVINO under torch.device("cuda") or set_default_device("cuda") failed on a CPU-only PyTorch with "Torch not compiled with CUDA enabled", and on a CUDA build it used the GPU for nothing. The pass evaluates everything else on the CPU already; this tensor now goes there too.

With inference_precision given twice, for example bf16 and then an empty value, lowering keeps the last entry and so compiles with the float32 default, but the runtime skipped empty entries and applied bf16. On x86, where the runtime decides, the program then computed in bfloat16. The runtime now takes the last entry as well, including an empty one, which means the default.

create_owned_output reported errors through its return value and its buffer through an out-parameter, and took a whole tensor only to read its dtype. It now returns exr::Result<ov_tensor_t*> and takes the dtype, like create_ov_tensor beside it, so its call site reads like the one for create_ov_tensor again. No behavior change.

…_precision The check that the runtime takes the last inference_precision entry ran only on x86 CPUs with bfloat16 units, so CI could skip it. A new test gives an invalid value and then an empty one. Lowering keeps only the empty one, and the runtime has to as well, or it fails to load the program. On the runtime before that fix it fails with "unsupported inference_precision 'f7'".

…ioner OpenVINO converts min, max, amin, amax, minimum, maximum, clamp and sign to its ReduceMin, ReduceMax, Minimum, Maximum and Sign operations, and none of those accept bool tensors. The partitioner still gave these ops to OpenVINO when they read bool tensors, so the whole program failed to lower: ReduceMin ... Element type of data input must be numeric. Got: boolean With PyTorch 2.15, MultivariateNormal exports two such mins when it checks scale_tril, so a model that builds one could not use OpenVINO. The partitioner now leaves these ops to the CPU when every tensor they read is bool, and the portable kernels compute them as PyTorch does. An op with a bool and a float input, such as maximum(mask, x), is computed in float, which OpenVINO converts, so it stays with OpenVINO. Other dtypes are not affected.

Lowering passes the precision too, so the comment was wrong to say only the runtime does. What makes the check depend on the runtime is that on x86 OpenVINO applies only the precision it gets when it loads the model.

The cumsum fold lines had landed between the comment and the decoder call it explains, so the comment read as if it were about the fold.

Since each cumsum is folded on its own, _evaluate is only ever asked for one node. Taking and returning that one value drops the list and dict it no longer needs, and a check that could never be false.

Only the attention mask test covered an int32 cumsum, and only through an OpenVINO CPU bug that newer OpenVINO releases fix. A fold limited to int64 now fails a pass-level test too.

Only x86 CPUs with bfloat16 units computed in bfloat16 before, so only they get slower, and only there does asking for bf16 bring the earlier speed back. On other x86 CPUs OpenVINO already computed in float32.

The precision docs said int64 results only go wrong outside the int32 range. Measured through the delegate on a CPU, an int64 sum is computed in float32: 16777216 + 1 gives 16777216, and 123456789 + 987654321 gives 1111111168. cumsum and prod near 2^24 stay exact.

Only an output compiled with a fully fixed shape refuses a buffer of another shape. A partly dynamic output accepts an empty one. Comment only.

OpenVINO 2025.1 has the same CPU plugin bug with integer cumsums over constants as the later releases, so the fold's docstring now starts the range there. Docstring only.

The OpenVINO CI job runs the op tests and then the model tests under set -e. Now that the test runner exits with code 1 when a test fails, a failing op test stopped the job before the model tests started, so a red run said nothing about the models. The script now runs both suites and exits with code 1 if either failed.

shoumikhin and others added 30 commits October 29, 2026 16:09

Merge release/1.6 into the #24222 cherry-pick

d22b428

Say why the OpenVINO bf16 addmm check needs the runtime

872b166

Lowering passes the precision too, so the comment was wrong to say only the runtime does. What makes the check depend on the runtime is that on x86 OpenVINO applies only the precision it gets when it loads the model.

Keep the OpenVINO input types comment above the converter call

1cd3643

The cumsum fold lines had landed between the comment and the decoder call it explains, so the comment read as if it were about the fold.

Evaluate one node at a time in the OpenVINO cumsum fold

63d0198

Since each cumsum is folded on its own, _evaluate is only ever asked for one node. Taking and returning that one value drops the list and dict it no longer needs, and a check that could never be false.

Test the OpenVINO cumsum fold on an int32 cumsum

0dd8ef6

Only the attention mask test covered an int32 cumsum, and only through an OpenVINO CPU bug that newer OpenVINO releases fix. A fold limited to int64 now fails a pass-level test too.

Say which x86 CPUs the OpenVINO float32 default slows down

06bee66

Only x86 CPUs with bfloat16 units computed in bfloat16 before, so only they get slower, and only there does asking for bf16 bring the earlier speed back. On other x86 CPUs OpenVINO already computed in float32.

Say which OpenVINO outputs refuse an empty buffer

1623b2e

Only an output compiled with a fully fixed shape refuses a buffer of another shape. A partly dynamic output accepts an empty one. Comment only.

Say the OpenVINO cumsum fold is needed from 2025.1

a19225d

OpenVINO 2025.1 has the same CPU plugin bug with integer cumsums over constants as the later releases, so the fold's docstring now starts the range there. Docstring only.

pytorchbot temporarily deployed to cadence November 2, 2026 17:02 — with GitHub Actions Inactive

This was referenced Nov 2, 2026

[v1.6.0] Release Schedule and Tracker #23608

Open

OpenVINO: LeRobot policies that do not run yet, and what blocks them #24325

Open

pytorchbot temporarily deployed to cadence November 2, 2026 17:42 — with GitHub Actions Inactive

pytorchbot temporarily deployed to cadence November 2, 2026 17:45 — with GitHub Actions Inactive

pytorchbot temporarily deployed to cadence November 2, 2026 17:48 — with GitHub Actions Inactive

shoumikhin changed the base branch from cherry-pick-24294-by-pytorch_bot_bot_ to release/1.6 November 2, 2026 17:58

shoumikhin requested review from JacobSzwejbka, SS-JIA, digantdesai, larryliu0820, metascroy, psiddh, rascani, robert-kalmar and shoumikhin as code owners November 2, 2026 17:58

JacobSzwejbka closed this Nov 2, 2026

JacobSzwejbka reopened this Nov 2, 2026

github-actions Bot added ciflow/trunk module: arm Issues related to arm backend labels Nov 2, 2026

JacobSzwejbka temporarily deployed to cadence November 2, 2026 18:28 — with GitHub Actions Inactive

JacobSzwejbka temporarily deployed to cadence November 2, 2026 18:44 — with GitHub Actions Inactive

JacobSzwejbka temporarily deployed to cadence November 2, 2026 18:46 — with GitHub Actions Inactive

JacobSzwejbka temporarily deployed to cadence November 2, 2026 18:49 — with GitHub Actions Inactive

JacobSzwejbka deployed to upload-benchmark-results November 2, 2026 20:35 — with GitHub Actions Active

Navigation Menu

Repository navigation

OpenVINO: fix bool min/max and masked read lowering, and follow-ups to #24294 - #24446

OpenVINO: fix bool min/max and masked read lowering, and follow-ups to #24294#24446
pytorchbot wants to merge 35 commits into
release/1.6from
cherry-pick-24396-by-pytorch_bot_bot_

pytorchbot commented Nov 2, 2026 •

edited by pytorch-bot Bot

Loading

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pytorchbot commented Nov 2, 2026 • edited by pytorch-bot Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is wrong today

What this change does

What was tested

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pytorchbot commented Nov 2, 2026 •

edited by pytorch-bot Bot

Loading