Repository navigation
Conversation
…ls (Step-5)
New --hybrid-swa-hits mode for the standalone indexer (default off; other
models unchanged). For engines that interleave sliding-window groups with full
attention and run MTP as EAGLE (Step-5-Preview: 4x SWA-512 + 1 full-attention
group, block 64), the FA-only, set-semantics tree mispredicts the engine's
prefix hit (DEEP-309: continuations 0/20 within 64 tokens, first reuse of a
new scaffold junction over-predicted by 3.4K-30.7K, 9-12% under-prediction
after eviction).
The mode adds a per-worker oracle fed from the same ZMQ stream:
- per-copy refcounts for every group; a Removed for a block with another
cached copy is held back from the radix tree
- sliding-window groups (filtered by the main path) are tracked by engine
block hash, which vLLM shares across groups
- /query reports min(tree, oracle) per worker, where the oracle predicts
max{k <= min(chain, (n-1)/bs) : every SWA group holds [k-need, k)} - mtp,
need = ceil((window-1)/bs) + mtp (9 for Step-5)
Flags: --hybrid-swa-hits, --mtp-drop-blocks (default 1), --swa-need-blocks.
Port of the reference model that matched engine cached_tokens on 56/56 parity
and 60/60 eviction probes in DEEP-309.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
documentation
deepapi's KV routing calls /query_by_hash with xxh3 local block hashes it computes itself (seed 1337, the indexer's XXH3_SEED), not /query. With --hybrid-swa-hits the by-hash path now reports min(tree, oracle) per worker too. The probe hashes only the full blocks of a longer prompt, so the oracle treats every hashed block as reusable (n = blocks * bs + 1); with n = blocks * bs it would under-predict one block for the 63/64 of prompts that are not an exact block multiple. The override is factored into apply_hybrid() and covered by 7 by-hash tests of the DEEP-309 shapes (resend, new scaffold junction, continuation, duplicate-copy eviction, never above the tree, unknown worker). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
temirulan
had a problem deploying
to
external_collaborator
October 15, 2026 21:55 — with 
GitHub Actions
Failure
GitHub Actions
Failure
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opt-in
--hybrid-swa-hitsmode for the standalone KV indexer. It makes/querypredict the prefix hit a hybrid-attention engine actually serves. It is off by default: with the flag unset, nothing new is constructed, and the listener and query paths are unchanged for MiMo, Qwen and other models.Why
DEEP-309 (deepinfra/backend#6255) tested Step-5-Preview: 4x SWA-512 groups plus 1 full-attention (DSA) group, block 64, MTP run as EAGLE. The full-attention-only, set-semantics tree mispredicts its hits:
After eviction it under-predicts by 9–12% of served cached tokens. The engine re-stores duplicate copies, and evicting the older copy drops a block that is still cached.
The cause is structural:
A reference model fed from the same event stream matched the engine's
cached_tokenson 56/56 parity probes and 60/60 eviction probes.What
standalone_indexer/hybrid.rsholds a per-worker oracle fed from the same ZMQ stream:Removedfor a block that still has another copy is held back from the radix tree./queryand/query_by_hash: both reportmin(tree, oracle)per worker. The oracle returnsmax{k ≤ min(chain, (n−1)/bs) : every SWA group holds blocks [k−need, k)} − mtp, withneed = ceil((window−1)/bs) + mtp, which is 9 for Step-5./query_by_hashis deepapi's routing path. deepapi hashes the full blocks of the prompt with xxh3, seed 1337, which is this indexer'sXXH3_SEED. The oracle therefore treats every hashed block as reusable:n = blocks·bs + 1.Flags:
--hybrid-swa-hits--mtp-drop-blocks(default 1)--swa-need-blocks(an override; it is derived per group by default)--hybrid-swa-hitsrequires--block-size.Tests
cargo test -p dynamo-kv-router --features standalone-indexer --libpasses 606/606: 7 new tests inhybrid.rsand 7 new by-hash tests inserver.rs(apply_hybrid).apply_mainholding back the tree removal, full coverage, and the no-SWA cap.Not done
Offline parity replay. DEEP-309 did not capture raw event streams, only per-probe outcomes. The live gate is to re-run deepbot's
tools/probes.pyandtools/evict2.py(on di-slc6-61, under/data/deepbot/step-5-preview-deep-309/) against this image in shadow.Engine-side
/tokenizeandreasoning_content. DEEP-309 also found that/tokenizein chat form dropsreasoning_content. That needs a separate engine-side change and is out of scope here.Image:
localhost:30500/dynamo-indexer:kvtest-9bdca6760dEngine: KV events must be on (
--kv-events-config).VLLM_KV_EVENTS_USE_INT_BLOCK_HASHESalready defaults to 1 in the Step-5 image. The indexer only decodes int hashes, so never set it to 0.🤖 Generated with Claude Code