Repository navigation
Conversation
Run each partitioner’s edge transforms after caller-supplied passes so backends can use the standard lowering entry point. QNN contributes its existing transforms through the new hook.
Reserve names for constant tensors already present in the exported program before lifting new constants, so methods with folded tables retain distinct constants.
Choose the registered IOManager and its logits, method names and step sizes, rejecting unknown contracts and invalid metadata. Load and reset the selected methods; route prompt tails to decode with a side-effect-free capacity check. Preserve module and I/O-buffer lifetimes and expose a backend-neutral allocator hook.
Route backend options through QnnPartitioner.from_options and register an allocator that supplies QNN shared-memory buffers through the generic IOManager hook. The LLM runner remains backend-neutral.
Encode graph output positions and restore their order at both runtime tensor-allocation entry points. Preserve context-binary order and legacy mutable outputs, and ignore malformed or overflowing position labels without throwing. This runtime change needs Qualcomm review.
Move backend-neutral static-cache rewriting into the LLM extension and keep QNN partitioning and tuning in the backend. Detect sliding layers from model configuration, keep cache writes in place with standard reinplace, and register a versioned runner that handles fixed chunks, padding, counters and reset. Quantized caches, sha attention and full-length sliding mode are excluded.
Use authoritative positions, float-cache version and dtype checks, exact buffer geometry and selected methods. Bind optional HF state outputs to existing buffers, preserve input aliases and register each concrete IOManager from its own implementation. Float-only fixtures cover reset, padding, malformed metadata and near-context tails.
Define fixed methods, cache metadata, padding, absolute counters and state-output bindings, and link the contract from the Qualcomm guide. State float-cache support and CPU-tested sliding correctness without claiming unmeasured device performance or a zero-copy Python host API.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24573
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 6a85ef6 with merge base 5f72739 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This was referenced Nov 3, 2026
meta-cla
Bot
added
the
CLA Signed
This PR needs a
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Document the version-1 hf_static I/O contract
HF's per-step tensors do not make a fixed prefill / decode program a drop-in replacement for every HF runner. Define the actual contract: method names, inputs / outputs, cache metadata, absolute counters, padding / mask semantics, optional returned state, reset and runner responsibilities.
The Qualcomm guide points to the contract and describes float-only cache I/O, split / concat auto attention and the registered Python runner. Performance claims and experimental quantized-cache / sha / full-sliding options are removed. Sliding-window correctness is CPU-tested; device support / latency past context 512 is not claimed. Python native returned-output cloning is separate from the persistent driver buffers, so no zero-copy host claim is made.
Review order:
docs/source/llm-hf-static-io-contract.md, its link fromhf_static_io_manager.h, the Qualcomm guide, then the documentation toctree. This is PR 8 of the Stack 2 series. Its local parent isstack2/pr07-hf-static-runtime; its GitHub base ismain, with earlier stack commits included.Test Plan
Documentation source links / toctree and the no-experimental-options / no-device-claim boundary are checked locally. Full Sphinx documentation build is not available in the existing environment (sphinx / myst_parser not installed); no build-green claim is made for it.
Final top validation:103 Python tests +42 subtests passed; collision1, existing QNN hooks3 and adjacent units60 passed (167 tests total). C++223 passed +2 pre-existing skips, QNN host7/7 passed. Local ufmt/flake8 on31 Python files and clang-format on21 C++ files are clean. No runtime code changes are introduced by this PR.
Follow-ups
The IOManager does not expose cache inspection, adoption or fork. A future cache-ownership API needs an explicit contract version. HF eager sliding counters may stop at the window while exported counters remain absolute; the document warns against transferring state by copying those counters alone.
#23563, #23508, Ilyas's #23183 and #23583 remain listed external / related work, not bundled patches or merge-state claims. Qualcomm review / device-suite validation is still needed for the earlier output-order PR.
Stack (review in order)
Each PR includes the commits of the ones above it; review only its own top commits.