[mergennachin]

Summary

When two independent requests share a long prompt, the second can repeat prefill even though the first already holds the matching KV state. This adds an experimental serving path that clones the already-computed part of an active request's prompt, then prefills only the recipient's remaining input.

A: [shared prompt: already computed][A's own work] ---> continues
                 |
                 +--- clone committed state
                 |
B: [shared prompt: reused         ][B's own input] --> generates

For each arriving anonymous text request, the server selects the active request offering the longest usable match. The clone position is the minimum of the common prompt length, the donor's committed position, and the recipient's prompt length minus one. That last token still runs through the model to obtain the recipient's next-token logits. Generated donor output is excluded from matching.

The server makes one asynchronous clone attempt using state already computed; it does not wait for the donor to reach a future position. The runner completes the clone outside its locks and returns it through the serving control loop. Both requests then own independent sessions and generation settings. If cloning fails, including when sliding-window state has been overwritten, the recipient falls back to the existing cached or full-prefill path.

--active-prefix-reuse-min-tokens sets the minimum extra prefill tokens saved over a cached match. 0 disables the feature by default; the current experiment uses 1024. An equally useful cached prefix takes precedence. The policy uses the shared serving/runner code and existing MLX/CUDA clone implementations, with no scheduler or backend storage changes. Named sessions and multimodal requests retain their existing behavior.

For review, start with candidate selection and clone completion in serving_runtime.cpp, then the callback overload in runner.cpp, followed by the focused tests and launcher plumbing. This PR is independent of #24304.

This remains a draft while the paired replay and CUDA-device validation are pending. Authored with assistance from Codex.

Test plan

The rebased commit cf4a0cf633 was rebuilt from source, including the core runtime and MLX backend. It includes merged PRs #24302 and #24303.

• Native runner, serving, prefix-cache, and scheduler suites: 460 tests passed, including clone success/failure, cache preference, cancellation, shutdown, and automatic capture sizing.
• MLX cache suite: 141 passed, 2 existing skips; includes flat/SWA clone isolation and retention checks.
• Native Qwen3-0.6B and Muse Glimmer smoke tests on the rebased build: actual live reuse of 1,890/1,891 prompt tokens, with recipient output token IDs identical to fresh prefill.
• python -m pytest examples/llm_server/python/tests/test_serve_launcher.py examples/models/muse-glimmer/tests/test_serve_batching.py examples/models/muse-glimmer/tests/test_serve.py examples/llm_server/python/tests/test_sampling_params.py -q — 268 passed.
• Before rebasing: 18 active-prefix tests repeated 25 times; runner and serving suites also passed under ThreadSanitizer with no reports.
• Changed C++ files pass clang-format --dry-run --Werror; changed Python files pass Ruff E/F checks and formatting checks. Launcher complexity check passes after consolidating range validation.
• Paired AgentX-mini replay on the rebased build: 20 complete sessions, 684 calls per pass, active reuse disabled versus a 1,024-token threshold. Both passes use automatic capture sizing (8 capture slots), root concurrency 1, and 8 inference slots. Runs are serial; performance results pending.
• CUDA-device validation on an NVIDIA host.
• Full repository CI.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24471

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job, 1 Unrelated Failure

As of commit cf4a0cf with merge base 893d815 ():

CANCELLED JOB - The following job was cancelled. Please retry:

• pull / unittest / macos / macos-job (gh)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / nxp-mcuxpresso-test / linux-job (gh) (trunk failure)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.