Repository navigation
[executorch][native] Split engine host factories out of the PTN runners - #24356
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24356
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New FailureAs of commit 3379ad9 with merge base 0f8e096 ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
SS-JIA
left a comment
There was a problem hiding this comment.
Review automatically exported from Phabricator review in Meta.
…s 67 input features Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * __->__ #24355 Eight of the fourteen `test_linear.py` models were fed 64-feature inputs, but `Model` defaults to `in_features=67`, so eager rejected them and every backend skipped them. The dtype, no-bias, batch-size and leading-batch tests now pass 67-feature inputs. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254891 Pull Request resolved: #24355
e126415
into
gh/digantdesai/183/base
…se samples Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * __->__ #24357 * #24356 * #24355 Engines need op-level reference data that doesn't depend on any engine or device. `op_samples` writes one .ptn per op and quantization scheme, bundling `golden` (eager) and `precise` (float64, weights dequantized, activations unquantized) samples over the same inputs. Linear is the first op. Schemes are float, int4wo g32/g128 and 8da4w g32/g128, each with f32, f16 and bf16 methods; int8 weight-only is left out because no native engine runs it. Linear cases are the backend test suite's models, reused through a stand-in runner, plus decode, prefill, large-K, bias-free leading-batch, outlier-channel and zero-row cases. With CMake, the `native_op_samples` target writes them; it is not in ALL, so a plain build never needs torch. Ops and schemes are independent axes. An op is an `Op` in `ops()`, a function returning `Case`s plus the dtypes to record them in; a scheme is a `Scheme` in `schemes()` that decides which cases it fits, quantizes a model and decodes its weights. `generate` records every op under every scheme, one spawned process per package, and writes no package when a scheme fits none of an op's cases. `cases.py` holds the case type and the backend test suite adapter, `schemes.py` the quantization, and `record.py` the packaging, which knows neither torchao nor any op. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254877 Pull Request resolved: #24357
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * __->__ #24358 * #24357 * #24356 * #24355 Eager golden outputs differed by an ulp or two between generating hosts, because MKL and oneDNN pick kernels by ISA and MKL splits work by thread count. The `op_samples` genrule and the `native_op_samples` CMake target now pin MKL to its reproducible path, oneDNN and ATen to AVX2, and OpenMP and MKL to one thread. `MKL_NUM_THREADS` is set as well as `OMP_NUM_THREADS` so a host's own value cannot override it. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254894 Pull Request resolved: #24358
…samples Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * __->__ #24359 * #24358 * #24357 * #24356 * #24355 A fixed tolerance either hides engine bugs or fails legitimate numerics, so an engine is judged by its distance from `precise` relative to eager's. `OutputCheck` bounds RMSE and max error by multiples of eager's plus `rtol`, and the fitted absolute and relative bias likewise, plus `sigmas` standard errors. Non-finite values must equal golden. CMake builds it and its test too. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254878 Pull Request resolved: #24359
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * __->__ #24360 * #24359 * #24358 * #24357 * #24356 * #24355 The bundled samples need a harness that runs on any engine. `sample_test_main` registers one gtest per package and method and judges each output with `OutputCheck`; unsupported ops, dtypes and shapes fail. `ggml_sample_test{,_cuda,_hip}` and `vulkan_sample_test` run it on `op_samples`. With CMake, `native_ggml_sample_test` (CPU) and `native_vulkan_sample_test` depend on `native_op_samples` and run it on its output. The sample tests are manual for now, since gaps fail: `ci.skip_test` in Buck and `DISABLED` in CTest. The README says how to run them and what the generator needs. `vulkan_sample_test` runs on SwiftShader like `vulkan_engine_test`; preloading a driver's `libvulkan.so.1` runs it on that driver. The ggml CUDA targets preload the CUDA driver stub through their env, so `test_ggml_ops.py` no longer puts the runner's directory on `LD_LIBRARY_PATH`. Authored with Claude Code. **NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D128254880/)! Differential Revision: D128254880 Pull Request resolved: #24360
… activation quant is allowed Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * __->__ #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 ggml quantizes the activations of its quantized matmuls, so weight-only int4 and GGUF linears silently ran with numerics the PTN doesn't describe. Compile now rejects them unless `GgmlEngineOptions::allow_activation_quant_for_weight_only` is set. `ggml_ptn_runner` and `run_hf_llm_ggml` expose it as `--allow_activation_quant_for_weight_only`, which GGUF LLMs on ggml now need. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254887 Pull Request resolved: #24361
…t8-activation golden Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * __->__ #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 With activation quantization allowed, ggml's weight-only linears no longer compute what `golden` describes. Weight-only packages now carry `golden-int8_act`, eager with int8 per-token dynamic activations, which `sample_test_main` judges against under `--allow_activation_quant_for_weight_only`. The flag is a `HostOptions` field that Vulkan rejects; new `ggml_sample_test_act_quant` targets and the `native_ggml_sample_test_act_quant` ctest set it. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254888 Pull Request resolved: #24362
…ple tests Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * __->__ #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 By default ggml-CUDA's cuBLAS GEMMs compute in f16 for f16 and dequantized weights, and in TF32 for f32 ones, so the linear samples measured those defaults rather than ggml's kernels. The CUDA sample-test targets set `GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32` and `NVIDIA_TF32_OVERRIDE=0`; llama.cpp's defaults, `test_ggml_ops` and the bundle are unchanged. Authored with Claude Code. **NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D128254889/)! Differential Revision: D128254889 Pull Request resolved: #24363
…values Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * __->__ #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 Rounding toward zero when narrowing f32 to f16, which Vulkan and WebGPU allow, biases every stored value toward zero, and the bias compounds across layers. The README suggests rounding to nearest, ties to even, as eager does. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254885 Pull Request resolved: #24364
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * __->__ #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 The hand-written cases cover few shapes. `facto_cases` turns the argument tuples FACTO draws for an op spec from a fixed seed into one-sample cases, `facto_<i>`, and `linear.py` keeps every fifth `linear.default` tuple. The spec lives here, so ET backend tests are unchanged. The `native_op_samples` CMake target puts the vendored FACTO submodule on `PYTHONPATH` and reruns when its sources change. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254881 Pull Request resolved: #24365
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * __->__ #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 PTN carries GGUF k-quant constants, but no op package exercised them. A `Gguf` scheme, as `gguf_q4_k` and `gguf_q6_k`, writes random valid blocks to a .gguf with gguf-py and loads them back as a checkpoint would. `golden-int8_act` now comes from per-token int8 pre-hooks on the weight-only model, which also work for GGUF, so `Scheme.weight_only` replaces `Scheme.act_quant`. FACTO also keeps the tuples whose input features fit 256-element blocks. Authored with Claude Code. **NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D128254890/)! Differential Revision: D128254890 Pull Request resolved: #24366
…samples Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * __->__ #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 `precise` used weights dequantized in the method's dtype, so f16 and bf16 references held already-rounded weights while engines decode in float32. The `Gguf` and `Torchao` schemes now decode their weights in float32 before the float64 math and raise on any other torchao weight. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254893 Pull Request resolved: #24367
…-block scales in the op samples Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * __->__ #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 GGUF samples used fixed positive power-of-two super-block scales, so engines' decoders never saw negative, rounded or subnormal ones. A model's i-th `nn.Linear` now takes the i-th scale family, and a new `scale_families` case gives each family its own output. A zero-d family waits behind a TODO. With q4_k's d at 0 every weight is -dmin·m, so a token's 32 outputs share one activation-quantization error (correlation about 0.74) and `OutputCheck`'s bias standard error, which assumes independent outputs, comes out about 4.9 times too small; with q6_k's d at 0 every weight is zero and the outputs are just the bias. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254896 Pull Request resolved: #24368
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * #24370 * __->__ #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 `linear_dq8ca_q4gsw` failed on rank-1 activations: `is_2d_matrix` rejected them, and `get_quantized_input_num_blocks` read the size at `ndim - 2`. Both now treat a 1-D tensor as a single row. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254882 Pull Request resolved: #24369
… 1 to 4 Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * #24371 * __->__ #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 `linear_dq8ca_q4gsw_tiled` strode each group's weight sums by `div_up(N, 4)` texels, but the sums are laid out `{groups, align_up(N, 8)}`, as both the ET-VK AOT pass (`compute_per_group_sums`) and Native's `FuseQuantizedLinear` write them. When `div_up(N, 4)` is odd, every group after the first read another row's sums; the shader now strides by `2 * div_up(N, 8)`, which equals the old stride for every other N. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254883 Pull Request resolved: #24370
…near gaps Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * #24372 * __->__ #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 The Vulkan engine leaves int4 weight-only linears unlowered when N isn't a multiple of 4, so op-sample cases with N of 1 and 43 fail. It also computes a linear on a rank-4 input wrong, while rank-3 inputs pass. `TODO.md` records both, with the fix for the first and where to look for the second. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254884 Pull Request resolved: #24371
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * #24373 * __->__ #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 Vulkan lets the device choose how stores to rgba16f images round, and devices that round toward zero bias f16 outputs; shader float controls don't govern that conversion. `gen_vulkan_spv.py` now wraps `imageStore` on writable rgba16f images in a round-to-nearest-even helper, and six shaders that declared such images by hand, two of them test shaders, now use the codegen. On devices that round toward zero, stores of 65520 and above now give inf, as the CPU conversion does, instead of 65504. The four texture shaders that already call `round_to_half_rte` on their outputs (`binary_scalar_texture`, `full_texture`, `reduce`, `scalar_tensor`) now round twice, with identical results; removing those calls is a follow-up. `aten._to_copy` from f32 to f16 textures is a `vkCmdBlitImage`, which the codegen doesn't reach, so `test_to_copy` keeps its one-ulp tolerance. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128254886 Pull Request resolved: #24372
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24405 * __->__ #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 `ggml_sample_test_act_quant` (CPU) is the one sample test that passes every package, so it leaves the manual set and runs in internal CI. The OSS ggml job now runs it as `native_ggml_sample_test_act_quant`, along with `native_output_check_test`. The rest stay manual with a TODO: plain `ggml_sample_test` fails the weight-only packages by design, and the Vulkan, CUDA and HIP ones have gaps. The OSS ggml job installs `gguf` for the op sample generator, and the workflow also runs on changes to FACTO and `test_linear.py`, which the samples draw from. Authored with Claude Code. @bypass-github-export-checks @bypass-github-pytorch-ci-checks @bypass-github-executorch-ci-checks @exported-using-ghexport Differential Revision: D128668496 Pull Request resolved: #24373
…samples Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * __->__ #24405 * #24373 * #24372 * #24371 * #24370 * #24369 * #24368 * #24367 * #24366 * #24365 * #24364 * #24363 * #24362 * #24361 * #24360 * #24359 * #24358 * #24357 * #24356 * #24355 Engines resize dynamic inputs between inferences, so the op samples now cover a shape changing mid-run. A case built with `chain` is exported with its `Dim`s and recorded as one sample of four calls. Each `Dim` starts mid-range, where export traces it, visits both bounds and returns, every other `Dim` in the opposite order. The harness already runs a sample's calls in order on one executable from one context, resizing inputs between them, so only its header comment changes. The new linear cases are the XNNPACK delegate's dynamic linear tests, each also followed by a ReLU: one and two batch dims, one `Dim` on two dims or two inputs, and sequential and parallel linears, with batch dims down to 1, which tracing mid-range keeps export from specializing. The cases every quantization scheme fits take 256 input features over up to 64 tokens: `dynamic_tokens` in 2-D and, with a static batch of one as in an LLM, in 3-D, two batch dims sharing one `Dim` of up to 16, and the sequential and parallel linears. Authored with Claude Code. **NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D128770754/)! Differential Revision: D128770754 Pull Request resolved: #24405
Stack from ghstack (oldest at bottom):
Drivers other than
ptn_runner_mainneed to link an engine without its runner binary. Each engine'smake_engine_host()moves into a*_engine_hostlibrary andPtnRunner.hintoengine_host; the runners are unchanged.Authored with Claude Code.
Differential Revision: D128254879
NOTE FOR REVIEWERS: This PR has internal Meta-specific changes or comments, please review them on Phabricator!
Differential Revision: D128254879