OpenVINO: LeRobot policies that do not run yet, and what blocks them #24325

We exported the LeRobot policies to ExecuTorch and ran them with the OpenVINO and TensorRT backends. This issue lists what still keeps policies from working on the CPU backends (OpenVINO, XNNPACK, Qualcomm, Core ML, MLX), so it can be picked up. The GPU side (CUDA and TensorRT) is tracked in #24353.

Asks by party

• ExecuTorch (OpenVINO, XNNPACK, portable kernels, Core ML, Qualcomm): review the open fixes below and pick the merged ones into release/1.6. Several gaps have no fix or owner yet, listed per backend.
• OpenVINO: items 1 to 7 below are in OpenVINO's PyTorch frontend or CPU plugin.
• coremltools: 3 conversion bugs, worked around in ExecuTorch for now.
• LeRobot: 2 policy options and 3 checkpoint or tolerance gaps, listed at the end.

Results

These numbers are from Oct 31, on an older stack. A refresh on the current stack is in progress.

Milliseconds per action chunk, batch size 1. Every number matches eager PyTorch on the same weights, except the 2 marked *.

• OpenVINO: Intel Xeon Platinum 8339HC server CPU, 16 threads, OpenVINO 2026.4.1, with the fixes in Fix 0-d tensors, dynamic output sizes and integer reductions, and default to float32 on CPU, in the OpenVINO backend #24294.
• TensorRT: NVIDIA H100, Jetson AGX Thor, Jetson Orin Nano 8 GB.
• On the Orin Nano, pi0 and xvla use a low-memory export, and pi05 and groot are split into several programs.
• - does not export or run yet (fastwam on Jetson Thor is not measured yet). x does not fit in 8 GB.

* tdmpc is sensitive to bf16 (eager PyTorch in bf16 also differs). pi0_fast in fp32 matches a float32 eager reference.

Already fixed or in review

• OpenVINO
  • Fix crashes, wrong outputs and lowering failures in several backends and in exir #24222 (merged; release/1.6 pick Fix crashes, wrong outputs and lowering failures in several backends and in exir #24298 open): crashes, bool-mask and bf16 inputs, stray assert nodes
  • Fix 0-d tensors, dynamic output sizes and integer reductions, and default to float32 on CPU, in the OpenVINO backend #24294 (merged; release/1.6 pick Fix 0-d tensors, dynamic output sizes and integer reductions, and default to float32 on CPU, in the OpenVINO backend #24377 open): 0-d tensors, float32 by default, data-dependent sizes
  • OpenVINO: fix bool min/max and masked read lowering, and follow-ups to #24294 #24396 (merged; release/1.6 pick OpenVINO: fix bool min/max and masked read lowering, and follow-ups to #24294 #24446 open): min, max, clamp and sign on bool inputs (gaussian_actor), masked reads, follow-ups to Fix 0-d tensors, dynamic output sizes and integer reductions, and default to float32 on CPU, in the OpenVINO backend #24294
• Portable kernels and the export flow
  • Keep Half and BFloat16 sums in float in portable matmul, var, cumsum and pooling kernels, and fix portable cat #24401 (merged; pick Keep Half and BFloat16 sums in float in portable matmul, var, cumsum and pooling kernels, and fix portable cat #24443 open): Half and BFloat16 sums kept in float (smolvla, groot)
  • Add a portable kernel for antialiased bicubic resize #24400 (merged; release/1.6 pick Add a portable kernel for antialiased bicubic resize #24447 open): antialiased bicubic resize (vla_jepa, wall_x)
  • Keep ops above 6 dims out of XNNPACK and fix quantized ops it leaves out #24380 (merged; pick Keep ops above 6 dims out of XNNPACK and fix quantized ops it leaves out #24445 open): ops above 6 dimensions kept out of XNNPACK
  • Fix mixed-dtype cat and stack in to_edge and the edge check #24381 (merged; pick Fix mixed-dtype cat and stack in to_edge and the edge check #24450 open): mixed-dtype cat and stack
  • Support 3-D convolution in the portable convolution kernel #24385: 3-D convolution in the portable kernels (lawam, lingbot_va)
  • Fix export of zero-stride constants and inputs, and of constants that delegates share #24404: zero-stride constants and constants that delegates share (wall_oss_05)
  • Limit the default thread pool to the allowed CPUs on Linux (not Android) #24395 (merged; pick Limit the default thread pool to the allowed CPUs on Linux (not Android) #24449 open): default thread pool sized to the allowed CPUs on Linux
• Core ML
  • Core ML: run ops that read uint8 and int8 inputs on the CPU, refuse ops coremltools gets wrong, and fix fills, tanh GELU, RMSNorm and repeated outputs #24317: uint8 and int8 readers on the CPU, ops coremltools gets wrong, fills, tanh GELU, RMSNorm, repeated outputs (fixes the xvla compile)
  • Fix wrong dtypes and values in Core ML conversion #24424: wrong dtypes and values in conversion
  • Refuse ops coremltools cannot convert in the Core ML partitioner, and say why #24425: refuse ops coremltools cannot convert, and say why
• Qualcomm
  • Qualcomm AI Engine Direct - Keep node names unique across the QNN and Samsung passes #24316: node names unique across the QNN and Samsung passes
  • Qualcomm AI Engine Direct - Keep the dtype of a recomposed RMSNorm #24428: keep the dtype of a recomposed RMSNorm (smolvla, eo1, vla_jepa)
  • Qualcomm AI Engine Direct - Decompose kept-whole ops that HTP leaves on CPU #24429: decompose kept-whole ops that HTP leaves on the CPU

What blocks OpenVINO

1. Boolean-mask indexing does not convert (groot, vla_jepa, lawam)

Conversion stops with Dynamic rank for aten::index input is not supported. Both policies use the Qwen3-VL DeepStack code in transformers, which writes and then reads the same masked rows on several layers in a row (_deepstack_process):

local_this = hidden_states[visual_pos_masks, :] + visual_embeds
hidden_states[visual_pos_masks, :] = local_this

On the second layer, the boolean mask reaches aten.index with a dynamic rank. The cause is in the FX decoder: TorchFXPythonDecoder.get_shape_for_value reads only node.meta["tensor_meta"], which is None after ExecuTorch's to_edge, and does not fall back to node.meta["val"] as get_found_shape already does. A fallback to meta["val"] fixes it; a small Qwen3-VL text model with DeepStack inputs then converts and matches PyTorch.

2. cumsum over a bool tensor wraps after 255 (pi0, pi0.5, pi0_fast)

Partly fixed. The pi0 family builds position ids with a cumulative sum over a bool padding mask (pi0, pi0.5):

position_ids = torch.cumsum(pad_masks, dim=1) - 1

PyTorch returns int64 here. The OpenVINO frontend keeps the narrow input type for the accumulation, so the count wraps after 255 and prompts longer than 255 tokens get wrong position ids. ExecuTorch now casts the input of cumsum and prod to int64 (#24294), but aten::cumsum over bool or integer inputs, and aten::prod over integer inputs, should follow PyTorch's type promotion in the frontend itself.

3. int64 reductions are computed in 32 bits on the CPU plugin

Even with the int64 cast from item 2, the CPU plugin of OpenVINO 2026.4.1 computes prod, cumsum and sum in 32 bits, so a result past the int32 range comes back wrong with no error. For example, prod of the int32 tensor [50000, 50000], cast to int64 first, returns -1794967296 instead of 2500000000.

4. Integer cumsum over constants is wrong on the CPU plugin

Partly fixed. Masks built from an int64 or int32 cumsum over a constant can come back wrong on CPU from the second inference on: they keep only what earlier calls allowed. We saw this on 2025.4.1, 2026.0.0 and 2026.4.1, but not on the nightly. The pi0 family builds its masks this way. ExecuTorch now computes these cumsums during lowering (#24294). Without that step, pi0.5 stopped matching eager PyTorch after a few action chunks. With it, pi0.5 matches on 16 action chunks in a row. The plugin bug remains in released versions.

5. HETERO devices

Lowering for HETERO:CPU crashes in export_model(), and for AUTO:CPU the same call fails with Not Implemented. Also, OpenVINO rejects set_property on a HETERO: device string (2025.1.0 to 2026.4.1), which is how the ExecuTorch runtime applies inference_precision. Setting the hint on each listed device works.

6. A failed run gives no reason

Partly fixed. Without #24294, molmoact2 lowered to 2 OpenVINO partitions plus CPU ops, then the first delegate call failed with CALL_DELEGATE execute failed at instruction 0: 0x1. With #24294 it runs and matches eager PyTorch. Failures in OpenvinoBackend::execute still do not log OpenVINO's own message (ov_get_last_err_msg()), and some, such as a failure to set an input or output tensor, return with no log at all.

7. Lowering is slow for large models (wall_x, flux3, lingbot_vla_v2)

wall_x now finishes, but lowering alone takes about 8.3 hours on this CPU, with other jobs sharing it (a 61,000-node graph, 12 OpenVINO partitions). lingbot_vla_v2 exports, but OpenVINO lowering does not finish in 12 hours in either precision. flux3 lowers for about 9 hours and then fails in conversion. AtenIndexPutReplacer fails on an aten.index_put whose indices are a list of tensors: while building the index it adds an int64 tensor to an int32 one, and OpenVINO rejects the mixed types:

While validating node 'opset1::Add Add_18585 (opset1::Add Add_18584[0]:i64[1,1,1,1],
  opset1::Unsqueeze aten_unsqueeze_copy_default_23/...[0]:i32[128,1]) -> (dynamic[...])'

Part of the time is ExecuTorch partitioning, which slows down sharply for very large graphs. A profile of the OpenVINO conversion step would show how much is on the OpenVINO side.

8. eo1 does not match eager

eo1 exports and runs on OpenVINO, but its bundle misses the export's accuracy check: 2.06 against a tolerance of 0.2. Not yet narrowed.

9. bf16 constants are rounded differently

A bf16 constant can come back rounded differently from PyTorch. Small effect, low priority.

What blocks XNNPACK

• XNNPACK takes no bf16 compute, so a bf16 policy's matmuls run on ExecuTorch's single-threaded portable kernels. These export and match eager, but one action chunk takes minutes: smolvla 17 min, groot 50 min, vla_jepa about 57 min, evo1 about 80 min. No owner yet.
• Some float32 bundles are slower than eager on the same 16 cores: xvla 0.8x, fineart_vla 0.7x, pi0_fast 0.4x. No owner yet.
• lawam: the bundle fails at run time inside the delegate: Propagating input shapes failed. No owner yet.
• A long Method.execute keeps the Python GIL, so other Python threads in the same process stall while it runs (seen with 2 XNNPACK bundles in one process). No owner yet.

What blocks Qualcomm (export only, checked on the x86 HTP emulator)

• smolvla: with Qualcomm AI Engine Direct - Keep the dtype of a recomposed RMSNorm #24428 every program now compiles, but the bundle misses its check on the emulator (2.55 against 0.2). Being narrowed.
• groot, pi05: loading fails with Can't create context from binary. Error 1002. This is the HTP memory limit, not an ExecuTorch bug: one session has about 3.75 GB of usable address space, and these policies have about 6 GB of fp16 weights. They need the large parts quantized (16a8w or 16a4w), or fewer programs loaded at once.
• wall_oss_05: after a 49 minute compile, the int64-to-int32 pass leaves an index with an invalid type. No small repro yet.
• fineart_vla: the int64 ops are now left on the CPU (Qualcomm AI Engine Direct - Keep the dtype of a recomposed RMSNorm #24428, Qualcomm AI Engine Direct - Decompose kept-whole ops that HTP leaves on CPU #24429). The HTP compile then needs about 5.2 GB, over the 3.7 GB limit, so like groot and pi05 it needs the large parts quantized.

What blocks Core ML

• gaussian_actor: when a partition returns a tensor next to a same-shape expand of it, Core ML hands the tensor back as zeros on the CPU and GPU compute units. The fix is in the next Core ML: run ops that read uint8 and int8 inputs on the CPU, refuse ops coremltools gets wrong, and fix fills, tanh GELU, RMSNorm and repeated outputs #24317 update.
• pi0 (float32): a Core ML delegate over 1 GiB is copied and hashed inside to_executorch, which costs about 3 times its size in memory, so the export needs about 61 GiB.
• wall_oss_05: the Core ML partitioner's pre-pass repeats the partition search and stalls (85 minutes with no conversion started).
• vla_jepa (float32): misses its check by 5e-3 on both the CPU and the CPU-and-GPU compute units, and crashes on the "all" compute unit. Not narrowed yet.
• coremltools: an index_put with a whole slice and a multi-entry index, rms_norm exported without its eps argument, and an integer full with no dtype are converted wrongly. Core ML: run ops that read uint8 and int8 inputs on the CPU, refuse ops coremltools gets wrong, and fix fills, tanh GELU, RMSNorm and repeated outputs #24317 keeps these on the CPU for now.

What blocks MLX

• eo1, wall_oss_05: .numpy() not supported for tensor subclasses. Seen on an older stack; needs a run on the current one.

Outside the backends

• dm05, wall_x, molmoact2: stock torch 2.15 cannot save or load these programs; the PyTorch fixes are tracked in TensorRT: LeRobot policies that do not run yet, and what blocks them #24353.
• fastwam: the published lerobot/fastwam_base checkpoint has no action head, so it does not load for export. A fix for its CPU float32 layers is ready but waits behind LeRobot's open pull request limit.
• lawam: fix(lawam): let the VLM-to-LAM head take bfloat16 hidden states on the CPU huggingface/lerobot#5214 lets its head take bf16 on the CPU. The OpenVINO blocker is then item 1.
• lingbot_va: it exports now, but the bf16 check fails on the CPU for the same reason as on the GPU (the check is stricter than rounding noise, see TensorRT: LeRobot policies that do not run yet, and what blocks them #24353). In float32 the check passes, and the export then stops in ExecuTorch's edge verifier.
• flux3: the published so101 checkpoint uses history conditioning, which the export does not serve yet. fix(flux3): hand F3_NATTEN_BACKEND to NATTEN's kernels huggingface/lerobot#5213 and a queued pull request handle the NATTEN kernels.
• smolvla: its VLM always loads in bf16 (no dtype option), so on Core ML and XNNPACK it runs on portable kernels.
• pi0 on XNNPACK (float32): the check gives 2.10e-05 against a tolerance of 2e-05.
• tdmpc: exports on OpenVINO and XNNPACK, but lerobot-rollout cannot feed it on the test robot yet.

cc @suryasidd @daniil-lyakhov @anzr299