[wwwind]

Summary

Extend executor_runner with an opt-in benchmarking mode to measure the performance impact of zero-copy model-boundary I/O for the Arm VGF backend.

The goal is to provide reproducible end-to-end inference measurements and quantify the overhead of input/output copies, host-memory imports, and VGF execution.

Changes

• Persistent aligned I/O buffers: Allocate caller-owned input and output buffers with configurable alignment and bind them to ExecuTorch tensors. Buffers and tensor metadata remain valid across repeated executions, avoiding per-iteration input preparation.
• Configurable benchmarking: Add warmup and measurement iterations, allowing the same model and input data to be benchmarked with zero-copy enabled or disabled.
• Detailed VGF statistics: Capture per-inference latency, input/output copy bytes and memcpy time, zero-copy bytes, host-memory import overhead, binding refresh time, VGF execution time, and import rejection/fallback reasons.
• JSON reporting: Export per-iteration measurements together with device information, model details, buffer sizes, and benchmark configuration for offline comparison.
• Minimal measurement overhead: Preallocate statistics storage and disable per-iteration logging during benchmarking.

The implementation is guarded by EXECUTORCH_VGF_IO_STATS and enabled at runtime through --vgf_perf_json. Existing executor_runner behavior remains unchanged when benchmarking is disabled.

Usage

Build the VGF runner with EXECUTORCH_VGF_IO_STATS=ON, then compare the two modes using identical model inputs.

Copy-based execution:

ET_VGF_ZERO_COPY=0 ./executor_runner \
  --model_path=model.pte \
  --inputs=input.bin \
  --print_output=none \
  --vgf_perf_json=copy.json

Zero-copy execution:

ET_VGF_ZERO_COPY=1 ./executor_runner \
  --model_path=model.pte \
  --inputs=input.bin \
  --print_output=none \
  --vgf_perf_json=zero_copy.json

By default, each run performs 20 warmup iterations and 200 measured iterations.

Benchmark mode currently requires static, tensor-only inputs/outputs, non-memory-planned graph I/O, and a plain PTE.

Motivation

This provides a consistent way to evaluate zero-copy I/O optimizations on real devices, identify remaining data-transfer overhead, and distinguish improvements in VGF execution from reductions in ExecuTorch/VGF boundary overhead.

It is particularly useful for investigating latency-sensitive models such as Neural Super Sampling (NSS), where eliminating boundary copies can have a significant impact on end-to-end inference latency.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24415

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit e625490 with merge base ab68460 ():

NEW FAILURE - The following job has failed:

• pull / android / run-emulator (gh)
  Warning: An error occurred while preparing SDK package Android Emulator: Error on ZipFile unknown archive.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[rascani]

Any reason we can't put this in the Arm runner since the changes are arm-specific?