Repository navigation

Long-context: rolling KeyDiff KV-cache compaction + chunked prefill - #686

Merged

jakmro merged 58 commits into

mainfrom

kv-compress-keydiff

Jun 8, 2026

Merged

Long-context: rolling KeyDiff KV-cache compaction + chunked prefill#686
jakmro merged 58 commits into
mainfrom
kv-compress-keydiff

Commits on Jun 3, 2026
- Support chunked prefill for text-only causal LLMs
  ncylich
  committed
  60ef7ba
Commits on Jun 4, 2026
- Add KeyDiff query-agnostic KV-cache compression
  ncylich
  committed
  d2a12bd
- Precompute RoPE cos/sin tables to fix long-context precision
  ncylich
  committed
  10cbac8
- Fix chunked-prefill sliding-window KV cache corruption
  ncylich
  committed
  135a584
- Make rolling KV compaction the default for causal LLMs
  ncylich
  committed
  6820973
- Compact global + re-rope sliding-window layers at compaction (engine-only)
  ncylich
  committed
  d84b08b
- Merge #687: chunked prefill for text-only causal LLMs
  ncylich
  committed
  b17c723
- Default chunked prefill for text qwen3 conversions
  ncylich
  committed
  80a129c
- Skip non-attention caches in KV compaction (fix LFM2/hybrid corruption)
  ncylich
  committed
  20b596a
- Merge origin/main into kv-compress-keydiff
  ncylich
  committed
  7427f59
- Re-rope KV-shared global source layers with global theta
  ncylich
  committed
  5463bd5
- Disable kv_compress flag on invalid rolling config
  ncylich
  committed
  de45aae
- Remove NIAH KV-compress test fixtures and consumer test
  ncylich
  committed
  5dbe288
- Trim excessive comments across KV-compress and rope-table changes
  ncylich
  committed
  f603b4c
- Replace standalone rope-table test with one integrated optimize-pass test
  ncylich
  committed
  7574363
- Drop redundant keydiff_score formula comment
  ncylich
  committed
  58842d2
- Rename test_kv_compress_free_functions.cpp to test_kv_compress.cpp
  ncylich
  committed
  0211413
- Shorten KV-compress comments
  ncylich
  committed
  28ab27e
- Hoist RoPE rotation table out of compaction row loops
  ncylich
  committed
  10a923b
- Shorten RopeRot comment
  ncylich
  committed
  db17e67
- Share one un-rope table across a compaction's keep-set scoring
  ncylich
  committed
  9f384c0
- Vectorize int8 dequant in KV compaction
  ncylich
  committed
  8a20978
- Renumber compaction via table composition (drop per-survivor trig)
  ncylich
  committed
  304b865
- Skip thinking-token cache strip on a compacted cache
  ncylich
  committed
  d16cf73
Commits on Jun 5, 2026
- Preserve special tokens across KV-cache compaction
  ncylich
  committed
  2a61035
- Tighten PR comments to non-obvious behavior only
  ncylich
  committed
  4c8127d
Commits on Jun 6, 2026
- Grow the KV cache on demand and move it across the prefill handoff
  ncylich
  committed
  4390d58
- Use a media-specific default auto-prompt in the chat test tool
  ncylich
  committed
  659c22f
- Support full-context cache transpilation
  ncylich
  committed
  8627813
- Size Qwen chunked-prefill caches from the full context too
  ncylich
  committed
  85da4e5
- Supply cloud-handoff args in the chunked-bundle-flags run test
  ncylich
  committed
  7f0e854
- Size LFM2 decoder_step cache from the full context too
  ncylich
  committed
  aed4967
- Carry max_position_embeddings in common graph meta for rope precompute
  ncylich
  committed
  7309a1f
- Don't assert precision in the cache move handoff
  ncylich
  committed
  6ef5056
- Drop comments the code already expresses
  ncylich
  committed
  eaabcf8
- Trim kv_compress.h comments to the load-bearing math
  ncylich
  committed
  553db06
- Bake rope tables only into the lowered graph, not the saved IR
  ncylich
  committed
  37e5af4
- Fix conv/recurrent cache handoff and guard MLA-dim compaction
  ncylich
  committed
  88cf4e7
- Preserve special tokens per-head across KV-cache compaction
  ncylich
  committed
  40a5547
- Don't pass stale per-head protect when tracking is disabled
  ncylich
  committed
  f8a9446
- Trim per-head-protect comments to one line each
  ncylich
  committed
  6cd45c8
- Drop per-head-protect comments that restate the code
  ncylich
  committed
  f849932
- Disable KV compaction for Gemma4 thinking mode
  ncylich
  committed
  19a657d
- Suppress compaction before the first-token fast path too
  ncylich
  committed
  5f864bb
Commits on Jun 7, 2026
- cleaned comment
  ncylich
  committed
  1f98ccf
- Shrink the KV cache after compaction to reclaim long-prefill capacity
  ncylich
  committed
  c391c53
- Trim shrink/suppression comments to the load-bearing ones
  ncylich
  committed
  59e8742
- Keep Gemma-4 thinking in the KV cache like LiteRT
  ncylich
  committed
  0bfe84d
- Persist Gemma-4 thinking in chat surfaces; rewrite thinking test
  ncylich
  committed
  524fe9c
- Fix review: clean OpenAI response content; drop stale v1 thinking test
  ncylich
  committed
  48f270d
- Remove stale iOS harness reference to deleted thinking test
  ncylich
  committed
  2b33371
- Remove channel-token config orphaned by the thinking-strip deletion
  ncylich
  committed
  dfbefff
- Reproduce Qwen thinking opener in history so KV cache reuses across turns
  ncylich
  committed
  ac780c9
Commits on Jun 8, 2026
- Merge remote-tracking branch 'origin/main' into kv-compress-keydiff
  ncylich
  committed
  2e8a525
- Add cache_context_length parameter to component specs functions
  jakmro
  committed
  ab128ba
- Shorten keepsets_from_fp16 comment to one line
  ncylich
  committed
  211e5a2
- Remove gemma4 context-scaling benchmark script
  ncylich
  committed
  4ea057e
- Consolidate gemma4 thinking tests into test_llm
  ncylich
  committed
  b374390

Commits

Commits on Jun 3, 2026

Commits on Jun 4, 2026

Commits on Jun 5, 2026

Commits on Jun 6, 2026

Commits on Jun 7, 2026

Commits on Jun 8, 2026