Update dependency cache_dit to v1.5.3

[renovate]

Note

This PR body was truncated due to platform limits.

This PR contains the following updates:

Package	Change	Age	Confidence
cache_dit	==1.1.10 → ==1.5.3

---

Release Notes

vipshop/cache-dit (cache_dit)

v1.5.3

Compare Source

Release Notes: cache-dit v1.5.3

Release Date: 2026-10-15
Comparison: v1.5.2...v1.5.3 (10 commits)
Type: Feature + bugfix release

Summary

v1.5.3 fixes four correctness bugs and adds one feature. A silent GQA corruption in the
Ulysses KV replication path (replicate_kv_sequence) could turn outputs into pure noise for
GQA models on 4-way context parallelism; torchao int8 quantization was unusable on Ampere
GPUs and on torchao 0.18; the DMD calibrator silently reused the last snapshot on CPU; and
the DBCache continuous-cache streak counter could run past max_continuous_cached_steps. The
FFPA attention backend now accepts attn_mask and runs on sm_89 (Ada) GPUs.

This release also unifies the CUDA (SVDQuant) wheels with the PyPI release: they are now the
same distribution, told apart by the local version +cu13 and served from our own index, so a
single command installs either build.

Affected Components

Component	Path	Change
Distributed (Ulysses/GQA)	src/cache_dit/distributed/core/_distributed_primitives.py	Per-window KV head selection at world == group size (fix #​1115)
Attention (backend registry)	src/cache_dit/attention/backends/register.py	Probe enable_gqa once at import instead of per dispatch
Attention (FFPA backend)	src/cache_dit/attention/backends/ffpa.py, src/cache_dit/_utils/utils.py	attn_mask support and sm_89 admission
Quantization (torchao)	src/cache_dit/quantization/torchao/quantize_ao.py	Import on pre-Ada GPUs; single granularity for int8 configs
Caching (contexts/manager)	src/cache_dit/caching/cache_contexts/cache_context.py, cache_manager.py	Continuous-cache streak accounting
DMD calibrator	src/cache_dit/caching/cache_contexts/calibrators/dmd.py	driver="gesvd" only on CUDA
Release tooling	tools/build_releases.sh, tools/release_workspace.py, tools/repair_wheel_platform_tag.py, tools/inspect_release_wheel.py, tools/make_index.py, tools/publish_index.sh	Unified CUDA channel, relocatable RPATH, artifact assertions, index generation and publish
Docs	docs/user_guide/INSTALL.md, docs/user_guide/ATTENTION.md, docs/user_guide/QUANTIZATION.md, docs/user_guide/LOAD_CONFIGS.md, README.md	Install command, FFPA matrix, release flow

Breaking Changes

None. The retired cache-dit-cu13 distribution installs the same cache_dit/ package as
cache-dit==1.5.3+cu13, so uninstall it before switching (see Packaging).

New Features

• FFPA attention backend: attn_mask and sm_89 support (#​1119): the FFPA backend now
  accepts attn_mask -- 1-D and 2-D masks are expanded to [B, 1, 1, Nkv], 3-D and 4-D masks
  pass through, and a masked call forces the HND route (the NHD fast path is mask-free by
  design). sm_89 (Ada) is admitted for fp16 and fp8; fp4 and the NHD fast path remain
  sm_120-only. 70 test cases cover mask parity against SDPA and the arch matrix.

• Unified CUDA wheels: cache-dit==1.5.3+cu13 from our own index (#​1120, release
  engineering): the CUDA build of this release is published under the cache-dit name with the
  local version +cu13, so installing it is one command and needs no second distribution name.
  See Packaging below.

Bug Fixes

• Fix silent GQA corruption in replicate_kv_sequence at world == group size (#​1115):
  gather_replicated_kv_for_local_q skipped the per-window KV head selection whenever the
  global KV head count equaled the local Q head count. For GQA models this is not an identity:
  Boogu (28 Q / 7 KV) with Ulysses world=4 gives every rank 7 local Q heads, coincidentally
  equal to the 7 global KV heads, so every rank skipped the GQA mapping, attended with the wrong
  KV heads and produced pure-noise outputs. The shortcut now only applies to global MHA.
  Regression test: tests/parallelism/test_ulysses_gqa_multirank.py.

• Fix dmd_svd_precision="high" silently falling back on CPU (#​1116):
  torch.linalg.svd(..., driver="gesvd") was passed on every device. PyTorch rejects driver=
  for non-CUDA inputs, and _dmd_fit_one swallowed the error and returned None, so on CPU the
  DMD calibrator reused the last snapshot without a word. The driver is now selected per device,
  matching svdquant/lowrank.py. Test: tests/api/test_dmd_svd_precision.py (all three
  precision levels on CPU).

• Fix the DBCache continuous-cache streak accounting (#​1114): the streak counter was not
  restarted after a compute step, non-adjacent cache hits did not reset it, and the cap was reset
  in redundant places, so a run could exceed max_continuous_cached_steps; DBPrune's streak
  limits are now applied per timestep. Regression tests in tests/api/test_refresh_context.py
  cover interleaved warmup, dynamic computation masks, both CFG orders and repeated cache checks.

• Fix torchao quantization on Ampere GPUs and on torchao 0.18 (#​1118): two defects.
  _basic_filter_fn / _fallback_filter_fn took quant_ctx: QuantizeAOContext = QuantizeAOContext(), a default built at import time whose quant_type defaults to
  float8_per_row -- so __post_init__ asserted an sm89+ GPU on import, and every torchao
  quant type, int8 and int4 included, failed on A100/A800 with "FP8 requires Ada or newer GPUs
  (>=sm89), but got (8, 0)"; on CPU-only hosts the import raised NotImplementedError. The
  context is now a required argument (every caller already passed it through partial), and
  requesting FP8 on such a GPU still fails with the same assertion. Second,
  int8_per_row/int8_per_tensor built Int8DynamicActivationInt8WeightConfig(granularity= (PerRow(), PerRow())), which torchao 0.18 rejects with "Invalid granularity type"; a single
  granularity applies to both activation and weight in 0.17 and 0.18. Test:
  tests/quantization/test_torchao_int8.py.

• Fix the CUDA wheels' RPATH (part of #​1120): the extension carried the build environment's
  lib directory (.../envs/py312/lib), a path that means nothing on a user's machine and a wrong
  library if it happens to exist there. The release tooling now rewrites it to $ORIGIN and
  refuses to publish a wheel that is not relocatable. The wheels published as cache-dit-cu13
  carry the old path, so this removes an existing wart rather than fixing a regression.

Packaging and Installation

The CUDA (SVDQuant) wheels are now the same distribution as the PyPI release, distinguished by
the local version of the wheel (1.5.3+cu13) and served from a static PEP 503 index:

pip install -U cache-dit --extra-index-url https://vipshop.github.io/cache-dit/whl/cu13/
pip install "cache-dit==1.5.3+cu13" --extra-index-url https://vipshop.github.io/cache-dit/whl/cu13/  # pinned

• The former cache-dit-cu13 distribution (last release 1.5.2) is retired. Both names ship the
  same cache_dit/ package, so uninstall the old one before switching:
  pip uninstall cache-dit-cu13.
• Wheels cover CPython 3.10-3.14, manylinux_2_38_x86_64 (the floor follows the GLIBC symbols
  the extension needs) and the sm_80, sm_86, sm_89, sm_120a, sm_121a architectures.
• Add our index as an extra index, not as --index-url: it holds cache-dit alone, with no
  dependencies, so PyPI has to stay the index those come from. uv searches extra indexes before
  the default one, so the command above resolves the CUDA build there too; naming PyPI as the
  extra index instead leaves uv considering the PyPI release alone.
• Install torch first, from the PyTorch index you normally use. The wheels link against a CUDA 13
  torch and cover CUDA >= 13.0.

Performance Improvements

• torch.compile graph breaks from the enable_gqa probe (#​1115):
  _supports_enable_gqa() ran inspect.signature on every attention dispatch, which dynamo
  cannot fold away (lru_cache wrappers, id()-based pseudo signatures for C builtins), so the
  probe stayed visible to the recorder. The answer cannot change within a process, so it is now
  resolved once at import: measured on fast-boogu (single card, B=1, flash-varlen backend),
  this removed the last source of graph breaks in the compile cell, 56 -> 0.

Documentation

• README.md: installation instructions, refreshed.
• docs/user_guide/ATTENTION.md: the FFPA mask and arch matrix.
• docs/user_guide/INSTALL.md: the CUDA install command, the migration off cache-dit-cu13, and
  a maintainer section covering the release flow (profiles, publish, verification).
• docs/user_guide/QUANTIZATION.md, docs/user_guide/LOAD_CONFIGS.md: install commands synced.

Dependencies

• No dependency version changes. torchao 0.17 and 0.18 are both supported.

Testing

• New regression tests: tests/parallelism/test_ulysses_gqa_multirank.py (multi-rank GQA
  replication), tests/api/test_dmd_svd_precision.py (SVD driver per precision), and expansions
  to tests/api/test_refresh_context.py (streak accounting) and tests/api/test_ffpa_backend.py
  (mask parity and arch matrix).
• Release verification: the full profile builds all five CPython wheels end to end
  (cp310-cp314, manylinux_2_38_x86_64, 13 cubins per architecture per arch set), the index
  publisher passes 23 synthetic cases (sha256 anchors, merge, re-release, refusals) and the
  resulting wheels install from a local index -- cache_dit.__version__ == 1.5.3+cu13,
  version_tuple == (1, 5, 3, 'cu13'), and import cache_dit._C_svdquant loads.

Contributors

Thank you to everyone who contributed to v1.5.3:

• DefTruth (@​DefTruth)
• Windyskr (@​Windyskr)
• chengyaoxue64-lab (@​chengyaoxue64-lab)
• Keshav Chhabra (@​kv-248)
• NgZiXin (@​NgZiXin)
• copilot-swe-agent[bot]

Full Changelog

v1.5.2...v1.5.3

Pull requests: #​1114, #​1115, #​1116, #​1118, #​1119, #​1120

v1.5.2

Compare Source

Release Notes: cache-dit v1.5.2

Release Date: 2026-09-07
Comparison: v1.5.1...v1.5.2 (5 commits)
Type: Hotfix release

Summary

v1.5.2 is a hotfix release focused on stability of the caching + distributed-parallelism
combination. It fixes a crash when --cache is combined with context parallelism
(--parallel ulysses/ring), a NameError in the image_reward metric, a layerwise-offload
correctness bug for models with tied (weight-shared) parameters, and a cache-setup failure for
multi-transformer pipelines. It also adds an extra_parallel_kwargs escape hatch for
model-specific context-parallel options and fixes async-Ulysses wrapper gating.

Affected Components

Component	Path	Change
Caching (DBCache blocks)	src/cache_dit/caching/cache_blocks/pattern_base.py	Fn-residual sharding under CP (fix #​1094)
Caching (adapter)	src/cache_dit/caching/cache_adapters/cache_adapter.py	Per-instance _is_cached check for multi-transformer pipes
Distributed (config/dispatch)	src/cache_dit/distributed/config.py, .../dispatch.py	extra_parallel_kwargs passthrough for transformer & controlnet
Async Ulysses	src/cache_dit/distributed/async_ulysses/*	Gate wrappers on ulysses_async; flux2 comm ops as eager boundaries for torch.compile
Layerwise offload	src/cache_dit/offload/layerwise.py	Dedupe tied weights across offload targets
Metrics CLI	src/cache_dit/metrics/metrics.py	Fix image_reward NameError
Tests	tests/api/, tests/quantization/	Regression tests for multi-transformer cache & tied-weight offload

Breaking Changes

None.

New Features

• extra_parallel_kwargs for model-specific context-parallel options (#​1110):
  ParallelismConfig now accepts an extra_parallel_kwargs dict (distinct from the deprecated
  parallel_kwargs, which maps to standard config fields). Entries are opaque to cache-dit and
  forwarded to _ContextParallelConfig.extra_kwargs on both transformer and controlnet dispatch,
  letting downstream model hooks read custom toggles

Bug Fixes

• Fix crash when --cache is combined with --parallel ulysses/ring (#​1113, fixes #​1094):
  the cache Fn-blocks residual (_get_Fn_residual) subtracted a full-sequence
  original_hidden_states from the already context-parallel-sharded hidden_states, raising a
  shape-mismatch error (e.g. 1280 vs 640, ratio == CP size). The CP-enabled guard only
  inspected diffusers' native _diffusers_hook registry, while cache-dit's own Ulysses/Ring
  runtime attaches split hooks to the vendored _cache_dit_hook registry, so the guard was always
  False for the built-in CP backend. The fix detects split hooks in both registries and
  re-shards the full-sequence tensor via the block's own split hook before subtracting (with an
  explicit shape guard and a safe proxy fallback). The non-CP path is unchanged.
  Verified on Wan2.2-T2V: --cache --parallel ulysses|ring runs with real cache hits (~2x
  speedup at 49 steps) and structurally-identical output.

• Fix image_reward metric raising NameError (#​1112): cache-dit-metrics image_reward ...
  always died with NameError: name 'compute_reward_score' is not defined because the call site
  unpacked the directory-level compute_reward_score which was never imported (only the unused
  compute_reward_score_img was bound). The correct helper is now imported and bound, matching
  the working clip_score sibling.

• Fix layerwise offload corruption for tied (weight-shared) parameters (#​1111): weight tying
  (e.g. lm_head.weight sharing embed_tokens.weight, Qwen3-4B) made two offload targets reference
  the same live parameter with independent pinned mirrors and onload/offload state machines — one
  target's CPU rebind poisoned the other's GPU residency bookkeeping, and async prefetch exposed
  half-copied tensors (torn embedding reads / device-mismatch crashes). Shared states are now
  owned by their first target (duplicates dropped as bookkeeping-only transports), owners are
  forced persistent, and prefetch residency bytes are recomputed after dedupe. Verified
  bitwise-equal to the GPU-resident baseline across sync / async / async+prefetch.

• Fix cache setup failure for multi-transformer pipelines (#​1102): CachedAdapter.create_context()
  set _is_cached on the pipe class while _context_manager is set on the instance. For
  dual/multi-DiT models that call enable_cache once per transformer over fresh
  FakeDiffusionPipeline instances sharing one class, the first setup flipped the class-level flag
  and every sibling was then reported as already cached (without its own context manager),
  asserting in create_context(). Context creation is now skipped only when that specific instance
  actually has a context manager. A regression test was added.

• Fix async-Ulysses wrapper gating (part of #​1110): all async Ulysses processor wrappers
  (flux, flux2, qwen_image, zimage, zimage_controlnet, ovis_image, longcat_image) entered the async
  path on ulysses_degree > 1 alone. Because the class-level patch is process-wide, a second
  instance of the same processor with CP enabled but ulysses_async=False (e.g. klein-ht and
  klein-fit in one worker) was forced onto the async path once any instance opted in. Wrappers now
  also require ulysses_async=True, restoring per-instance opt-in.

• Keep flux2 async comm ops as eager boundaries for torch.compile (part of #​1110): inline
  _All2AllComm send/wait() calls induced graph breaks that pushed surrounding fp8
  torchao Linear calls into nested resume frames, crashing dynamo ("Polyfill handler does not
  have a traceable function"). The comm operations are now kept as eager boundaries.

Performance Improvements

• No standalone performance work in this release; the cache + Ulysses/Ring combination restored by
  #​1113 delivers the expected DBCache speedups under context parallelism (~2x measured on
  Wan2.2-T2V at 49 steps).

Documentation

• No documentation changes in this release.

Dependencies

• No dependency version changes.

Testing

• New regression tests: tests/api/test_multi_transformer_is_cached.py (multi-transformer cache
  setup), tests/quantization/test_layerwise_offload.py (tied-weight offload parity).
• End-to-end verification on Wan2.2-T2V (A14B): --cache with --parallel ulysses and ring
  across 2 GPUs; cache-hit correctness and structural (non-garbled) output confirmed; non-CP path
  byte-for-byte unchanged.

Contributors

Thank you to everyone who contributed to v1.5.2:

• DefTruth (@​DefTruth)
• Tai An / Anai-Guo (@​Anai-Guo)
• jiangyon.ren (@​jiangyon-amd)
• copilot-swe-agent[bot]

Full Changelog

v1.5.1...v1.5.2

Pull requests: #​1113, #​1112, #​1111, #​1110, #​1102 · Issue fixed: #​1094

v1.5.1

Compare Source

🚀 Cache-DiT v1.5.1 Release Notes

Release Date: 2026-09-01
Comparison: v1.5.0...v1.5.1 (45 commits, 43 PRs, 7 contributors)

📋 Overview

Cache-DiT v1.5.1 is a feature release spanning ~2.5 months (2026-06-16 ~ 2026-09-01), covering 43 PRs. The headline feature is the FFPA CUDA attention backends (fp16/fp8/fp4) for Blackwell GPUs with native zero-copy NHD layouts. The release also adds 7 new model integrations (boogu-image, ernie-image, Joy-Image-Edit, AnyFlow & AnyFlow-FAR, Krea-2, Bria-Fibo, Cosmos3 Omni), a Sage3 Blackwell attention backend, torchao NVFP4 dynamic quantization, an alternating SVD refinement option for SVDQuant, the hyper-parameter-free FoCa calibrator, Context-Parallelism communication optimization for boogu-image, and Ascend NPU profiling support.

✨ Core Highlights

1. ⚡ FFPA CUDA Attention Backends

New attention backends built on ffpa-attn (Flash Prefill Attention), delivering quantized CuTe TMA kernels for sm_120 (Blackwell) GPUs — forward inference only, Context Parallelism compatible:

• --attn ffpa: CuTe TMA fp16/bf16 kernels
• --attn ffpa_fp8: FP8 quantized attention, defaults to the highest-precision quant config (int8 QK MMA + fp16 PV accumulation, per-thread Q/K + per-channel V scales on consumer GPUs)
• --attn ffpa_fp4: NVFP4 quantized attention
• --attn ffpa_fp8_per_block: per-block quantized FP8 variant (fastest, lower precision)

Fine-grained CLI controls: --ffpa-hybrid (hybrid fp16 early-rows stage, default off for every backend), --ffpa-hybrid-n-early, --ffpa-fp8-hadamard / --ffpa-fp4-hadamard (Hadamard pre-rotation). The enable_fp8 + enable_fp4 combination is explicitly rejected with a clear error.

Native NHD layouts (#​1104–#​1107): all CuTe families (fp16/fp8/fp4) now consume NHD ([B, H, N, D]) inputs natively — both packed and strided — through the fp8 persist-D tensor_layout fast path, eliminating the BHSD↔NHD permute + .contiguous() materialization overhead on every forward.

Performance (FLUX.1-dev, 2048×2048, single RTX PRO 5000, 28 steps; E2E latency / PSNR / SSIM vs BF16 baseline, measured before the NHD fast path landed):

Backend	Latency	Speedup	PSNR / SSIM
BF16 baseline	92.46s	1.00×	—
sage	81.09s	1.14×	40.2 dB / 0.986
sage3	80.73s	1.15×	31.4 dB / 0.869
ffpa (fp16)	93.24s	~1.00×	49.3 dB / 0.997
ffpa_fp8	83.64s	1.11×	32.7 dB / 0.919
ffpa_fp4	78.66s	1.17×	33.6 dB / 0.949

With 2×GPU Ulysses Context Parallelism: ffpa_fp8 48.42s, ffpa_fp4 45.89s. The fp16 row reflects the pre-NHD permute overhead, which the native NHD path (#​1104–#​1107) removes.

PRs: #​1098, #​1099, #​1100, #​1101, #​1103, #​1104, #​1105, #​1106, #​1107

2. 🧩 New Model Integrations

Seven new pipelines gain Cache + Parallelism support (cache adapters, CP/TP wiring, and cache_dit.generate CLI examples):

• boogu-image (#​1072) — plus a dedicated CP optimization: reduced KV all-2-all communication overhead for boogu (#​1082) and per-model attention backend dispatch (#​1083)
• ernie-image (#​1073)
• Joy-Image-Edit (#​1079)
• AnyFlow & AnyFlow-FAR (#​1081)
• Krea-2 (#​1074)
• Bria-Fibo (#​1075)
• Cosmos3 (Omni) (#​1088, community contribution by @​johnnynunez) — new Cosmos3OmniPatchFunctor (functor_cosmos3.py) reorders the MoT dual-stream contract so DBCache tracks the actually-denoised gen stream (und is quasi-static and must not drive residual-diff decisions). Measured on Cosmos3-Nano T2V (RTX PRO 6000 Blackwell, 189 frames / 35 steps): 480p 138.59s → 76.75s (1.81×) with DBCache rdt=0.20, 2.04× with torch.compile; 256p 1.53×. Requires diffusers main for Cosmos3OmniPipeline.

3. 🔥 Sage3 Blackwell Attention Backend

• --attn sage3 (#​1085): SageAttention-3 backend for sm_120 (Blackwell) GPUs
• Fixed the sage3 (sm_120) tensor layout (#​1086)
• Made sage3-attn compatible with torch.compile (#​1087)

4. 💎 Quantization Enhancements

• torchao NVFP4 dynamic quantization (#​1090): new quant types nvfp4 and nvfp4_weight_only on the torchao backend for Blackwell GPUs. torchao>=0.17.0 is now required for the quantization extra.
• Alternating SVD refinement for SVDQuant (#​1095, community contribution by @​Rudin6): new opt-in svd_refine_iters knob. Each round refits the low-rank factors against smoothed_weight - dequant(quantize(residual)) so the low-rank branch absorbs the residual quantizer's error pattern. Default 0 is bitwise-identical to the previous one-shot behaviour (verified across 72 configurations). Measured on 14 PixArt-Sigma layers with real calibration activations: 5 rounds cut weight error by 7.8–8.9% and layer output error by 17.4–18.2% for both INT4 and NVFP4.
• SVDQ W4A4 pad-to-128 (#​1084): pad SVDQ linear layers to a multiple of 128 so the W4A4 kernel covers shapes that are not tile-aligned.
• Fix: clamp torchao float8 zero scales to avoid 0/0 NaN (#​1097) — previously, all-zero scale rows produced NaNs on dequantization.
• Cleaner quantization logging format (#​1091)

5. 🔮 FoCa Calibrator

New FoCaCalibrator (#​1066): forecasts cached tensors with a BDF2 predictor + Heun corrector. It is a drop-in alternative to TaylorSeerCalibrator and DMDCalibrator with zero hyper-parameters — step intervals are derived automatically from the step counters.

6. 📈 Parallelism & Profiling

• CP communication optimization (#​1082): reduced boogu-image KV all-2-all communication overhead on the Context-Parallelism path.
• Ascend NPU profiling (#​1096, with @​changetheway and @​ooooooye): cache_dit.profiler now supports torch_npu.profiler for Ascend NPUs, mirroring the torch.profiler API.

🔧 Other Enhancements

• CLI: support extra input kwargs (#​1068); allow using local test data (#​1092); new cache+dmd+svdq YAML example configs (#​1069)
• Docs: Cache-DiT tech report added (#​1062); DMD calibrator design docs (#​1063, #​1064, #​1065); docs rendering fix (#​1067); SVDQ NVFP4 kernel comments (#​1071)
• Community: new community integration links in README (#​1108)
• Dev experience: agent workflows + model-integration skill updates (#​1074, #​1075, #​1076, #​1080); skills relocated to .github/skills (#​1109); example data updates (#​1077, #​1078)

🐛 Bug Fixes

• sage3 (sm_120) attention backend tensor layout (#​1086)
• torchao float8 zero scales clamped to avoid 0/0 NaN (#​1097)
• ffpa_fp8_per_block V quantization switched to per-channel for 2048-resolution precision (#​1103)

⬆️ Dependencies

• torchao>=0.14.1 → torchao>=0.17.0 (quantization extra; required for NVFP4 dynamic quantization)

👥 Contributors

Thanks to everyone who contributed to this release:

@​DefTruth, @​Rudin6 (first contribution), @​johnnynunez (first contribution), @​changetheway, @​ooooooye, Copilot

Full Changelog

v1.5.0: Major Release

Compare Source

🚀 Cache-DiT v1.5.0 Release Notes

Release Date: 2026-06-16
Comparison: v1.3.0...v1.5.0 (176 commits)
Full Changelog: GitHub Releases

📋 Overview

Cache-DiT v1.5.0 is a major feature release spanning 3 months (2026-03-12 ~ 2026-06-16), covering 176 PRs. This release is organized around four core modules: SVDQuant W4A4 Quantization (PTQ/DQ/NVFP4/Converter CLI), DMD Calibrator (exponential-basis Dynamic Mode Decomposition), Bucket-style Layerwise CPU Offload (compute-communication overlapped offloading), and Ray Wrapper (transparent distributed inference). Additional highlights include FLUX.2-klein-kv series support, full CUDA Graph integration, extensive kernel refactoring, quantization enhancements, parallelism framework improvements, and comprehensively updated documentation.

✨ Core Highlights

1. 💎 SVDQuant W4A4 Quantization

Cache-DiT v1.5.0 natively integrates a complete SVDQuant W4A4 quantization workflow — the most significant feature of this release. Instead of relying on third-party libraries, users can now perform end-to-end W4A4 quantization directly through Cache-DiT's cache_dit.quantize() / cache_dit.load() API.

📊 PTQ (Post-Training Quantization)

Supports svdq_int4_r{rank} and svdq_nvfp4_r{rank} quant types:

• INT4 PTQ (≥sm80): Collect activation statistics via calibrate_fn → SVD low-rank decomposition → INT4 packing → runtime W4A4 GEMM. Supports three calibration precision levels: low (recommended default, ~18× speedup), medium, high. Serialize to {quant_type}.safetensors + quant_config.json; restore via cache_dit.load().
• NVFP4 PTQ (≥sm120, Blackwell): Designed for RTX 5090 and other Blackwell GPUs. Currently only runtime_kernel="v1" is supported for NVFP4.

Performance (FLUX.2-klein-4B, 1024×1024, L20):

Stage	Latency (s)	Memory (GiB)	Transformer Weight (GiB)
BF16 baseline	2.13	17.32	7.22
SVDQuant INT4	1.24	12.39	2.28
SVDQuant + compile	1.02	12.39	2.28

• Transformer weight reduction: ~3.2× compression (7.22 → 2.28 GiB)
• End-to-end latency: ~1.7× speedup (2.13 → 1.24s), ~2.1× with compile (2.13 → 1.02s)
• PSNR > 29 dB, near-lossless visual quality

NVFP4 Performance (RTX 5090, FLUX.2-klein-4B, 1024×1024):

Stage	Latency (s)	Speedup	Memory (GiB)
BF16 baseline	0.97	1.00×	17.32
NVFP4 PTQ	0.58	1.69×	12.50
NVFP4 + compile	0.47	2.05×	12.50

⚡ DQ (Dynamic Quantization)

Zero-calibration quantization via _dq suffix types (e.g., svdq_int4_r128_dq):

• identity (default): Apply SVD low-rank decomposition directly to the original weight matrix — no calibration, no serialization.
• weight / weight_inv: Weight-statistics-only heuristic smooth strategies (experimental).
• few_shot: Collect a small number of real inference forwards at runtime, then quantize in-place with configurable relaxation strategies (7 strategies: auto/stable_auto/power/log/rank/top/fixed). Supports few_shot_auto_compile for deferred compilation after quantization.

DQ Performance (FLUX.2-klein-4B, 1024×1024, identity, rank=128): 1.28s, PSNR 28.71 dB.

🔧 SVDQ Converter CLI

New cache-dit-convert command-line tool for one-click model conversion to SVDQ W4A4:

cache-dit-convert --model-path black-forest-labs/FLUX.2-klein-4B \
  --save-dir ./FLUX.2-klein-4B-svdq \
  --quant-type svdq-int4-r128-dq

Supports INT4/NVFP4 DQ conversion, custom smooth strategies, and multiple precision options. Outputs {quant_type}.safetensors + quant_config.json.

🔀 Fused MLP

New fused_gelu_mlp / fused_gelu_proj passes (enable via svdq_kwargs["fused_mlp"]=True), fusing the first GEMM + GELU activation into a single kernel for reduced kernel launch overhead and intermediate activation memory.

🔗 Parallelism Compatibility (Cache-DiT Exclusive)

SVDQuant in Cache-DiT fully supports Context Parallelism (Ulysses / Ring / USP) and Tensor Parallelism (PyTorch DTensor). Users can layer quantization on top of distributed parallelism for extreme memory compression and throughput gains. SVDQW4A4ShardLinear (dtensor.py) provides native TP sharding support. This is a differentiating capability of Cache-DiT's SVDQuant implementation versus other W4A4 solutions.

⚙️ Quantization Configuration Enhancements

• Regional Quantization (regional_quantize=True + repeated_blocks): Quantize only repeated transformer blocks, keeping embedding layers at full precision.
• Hybrid Precision Plan (precision_plan): Assign different quant types to different sub-layers by name pattern.
• FP8 Per-Tensor Fallback (per_tensor_fallback=True, default on): Auto-fallback from per-row to per-tensor for TP-incompatible layers.
• TorchAO Backend Refactor: Cleaner QuantizeBackend enum (AUTO / TORCHAO / CACHE_DIT / NONE).
• Quantize API Refactor: Deprecated legacy kwargs, unified under QuantizeConfig + svdq_kwargs.

📦 cache-dit-cu13 Pre-built Wheel

Pre-compiled SVDQuant wheel for CUDA 13 users: pip install cache-dit-cu13 — no source build needed.

---

2. 💾 Bucket-style Layerwise CPU Offload

Cache-DiT v1.5.0 introduces a novel bucket-style layerwise offload mechanism that overcomes the "every layer waits for its own transfer" inefficiency of traditional approaches.

Core Design:

• Bucket Pipeline: Divide target modules into small contiguous buckets; prefetch the next bucket asynchronously while the current one executes.
• Dual Independent Copy Stream Pools: Separate CUDA stream pools for onload (H2D) and offload (D2H).
• Persistent Bins: Distribute the persistent budget evenly across the target sequence.
• Flexible Resource Controls: transfer_buckets, persistent_buckets, persistent_bins, prefetch_limit, max_copy_streams, max_inflight_prefetch_bytes.

Performance (FLUX.1-dev, L20):

Config	Memory	Latency
No offload	~38 GiB	23.4s
Diffusers sequential	~1 GiB	335s
Layerwise (transfer=4, persistent=32, bins=4)	~16 GiB	24.6s

Only ~1.2s added latency versus no-offload baseline for a 38→16 GiB memory reduction.

torch.compile Compatible: Apply offload before compile; offload hooks work correctly after compilation.

CLI Quick Start:

python3 -m cache_dit.generate flux \
  --layerwise-offload --layerwise-async-transfer \
  --layerwise-transfer-buckets 4 --layerwise-persistent-buckets 32 \
  --layerwise-persistent-bins 4 --layerwise-prefetch-limit \
  --layerwise-max-inflight-prefetch-bytes 8gib --compile

---

3. 🌩️ Ray Wrapper (Transparent Distributed Inference)

The Ray Wrapper makes distributed inference completely transparent to user code. No torchrun, no dist.init_process_group, no manual model sharding — just use_ray=True, and Cache-DiT handles everything.

Two Wrapper Levels:

Level	Description	Best For
Pipeline Wrapper (recommended)	Ray manages the entire pipeline execution	Full feature support (cache, quant, parallelism), simplest, fastest.
Transformer Wrapper	Only the transformer runs on Ray workers	Lightweight, but slight slower

Key Features:

• ray_transfer_fn: User-defined per-worker model loading, bypassing serialization overhead and custom module class resolution issues.
• ray_use_compile: Automatic per-worker compilation.
• ray_runtime_env: Custom module import handling via PYTHONPATH.
• Supports all parallelism strategies: TP, Ulysses, Ring.
• LoRA support: fuse before enabling (TP requires fused LoRAs).

Performance (FLUX.2-klein-base-9B):

Config	Latency
Baseline (single GPU)	47.41s
Ray TP=2 + compile	24.57s

Minimal Example:

cache_dit.enable_cache(
  pipe,
  parallelism_config=ParallelismConfig(tp_size=2, use_ray=True),
)
image = pipe(prompt="A cat...").images[0]  # Code unchanged

---

4. 🔮 DMD Calibrator (Dynamic Mode Decomposition)

DMD (Dynamic Mode Decomposition) is an exponential-basis feed-forward calibrator, serving as a drop-in alternative to TaylorSeer's polynomial basis.

Mathematical Principle: The cached feature stream is modeled as a linear dynamical system $Y_{t+1} \approx A \cdot Y_t$. The propagator $A$ is identified via one economy SVD, then eigendecomposed for extrapolation via $\lambda^k$. Unlike TaylorSeer's polynomial extrapolation (diverges as $t^n \to \infty$), DMD is bounded when $\lvert\lambda\rvert \leq 1$.

TaylorSeer vs DMD:

Aspect	TaylorSeer (Polynomial)	DMD (Exponential)
Basis	$Y(t) \approx \sum \frac{d^nY}{dt^n} \frac{t^n}{n!}$	$Y(t) \approx \Phi \cdot \text{diag}(\lambda^t) \cdot b$
Extrapolation	Diverges as $t^n \to \infty$	Bounded when $\lvert\lambda\rvert \leq 1$
Snapshots needed	2+ (1st order)	≥ 4 uniformly spaced
Best for	DiT-class denoising (DDPM)	Flow-matching generators (Hunyuan3D, etc.)
Noise sensitivity	Low	Moderate (SVD truncation suppresses noise)

Usage:

cache_dit.enable_cache(
  pipe,
  cache_config=DBCacheConfig(...),
  calibrator_config=DMDCalibratorConfig(
    dmd_history=6, dmd_rank=0, dmd_ridge=1e-8,
  ),
)

CLI: python -m cache_dit.generate flux --cache --dmd --dmd-history 6

---

🔧 Other Enhancements

🧩 FLUX.2-klein-kv Series

• TP + compile integration (#​888)
• fp8 per-row + TP support (#​896)
• Async Ulysses support (#​877)

📈 CUDA Graph

• Full CUDA Graph support (#​942-#​952)
• CUDA Graph + fp8 rowwise integration (#​952)
• descent_tuning enabled by default (#​935)
• compile full-graph CLI option (#​946)

🏗️ Kernel & Infrastructure Refactoring

• Triton kernel refactoring (3/N, #​907-#​909)
• Communication kernels registered as torch.library ops (#​905)
• CuTeDSL communication kernels (fp8, #​977), merge-attn-states kernel (#​973)
• Unified all2all/ring communication API (#​985)
• Async Ulysses refactoring (#​986)
• Unified ops registration policy (#​931, #​939, #​967)

🔗 Parallelism Improvements

• record_plans for TP/TE-P planners (#​916-#​917)
• sub cp_plan support (#​989)
• CP & VAE-P planner refactoring for better logging (#​918)

⚡ Compilation Optimizations

• Removed manual graph breaks (#​885)
• CUDA Graph for dynamic compile (#​951)
• SVDQuant + compile compatibility; offload + compile (#​1014)
• Log suppression improvements (#​934, #​923)

🤝 Community Integrations

• MindIE-SD: Huawei Ascend NPU attention/compilation backend support (@​blian6, #​1004)
• TensorRT-LLM community link (#​991)

📦 Dependency Updates

• PyTorch → 2.11.0 (#​900)
• uv for dependency management (#​992)
• Python version compatibility fix (@​FNGarvin, #​1025)
• TorchAO ≥ 0.17.0

📝 Documentation

• New/reworked docs: QUANTIZATION.md, OFFLOAD.md, RAY.md, CACHE_API.md (DMD), CUDA_GRAPH.md, COMPILE.md
• Formatting and typo fixes (#​886-#​887, #​899, #​903, #​913)
• FAQ update: Flash Attention 2 install guide (#​915)
• Community link fixes (#​928, #​940)

⚠️ Breaking Changes

Change	PR	Migration
Serving module deprecated	#​933	Migrate to SGLang Diffusion or vLLM-Omni
Native Diffusers parallelism backend deprecated	#​1017	Use Cache-DiT native parallelism backends (better performance)
Quantize API legacy kwargs deprecated	#​910-#​911	Migrate to unified QuantizeConfig + svdq_kwargs. Legacy grad_ckpt, reorder_before_quantize etc. removed

👥 New Contributors

Thank you to the new contributors who joined the Cache-DiT community:

• @​blian6 — MindIE-SD NPU attention/compilation backend support
• @​FNGarvin — Python version compatibility fix
• @​Archerkattri — DMD Calibrator

🙏 Contributors

Thank you to all contributors for this release: @​DefTruth, @​Archerkattri, @​FNGarvin, @​blian6.

For the full list of changes, see GitHub Release v1.5.0 and the full changelog.

---

🇨🇳 中文版

📋 概述

Cache-DiT v1.5.0 是一次重大功能更新，历时 3 个月（2026-03-12 ~ 2026-06-16），涵盖 176 个 PR。本次发布围绕四大核心模块：SVDQuant W4A4 量化（PTQ/DQ/NVFP4/转换器CLI）、DMD Calibrator（基于指数基底的动态模态分解校准器）、Bucket-style Layerwise CPU Offload（计算-通信重叠的逐层卸载）、Ray Wrapper（分布式推理透明化包装器）。此外还包括 FLUX.2-klein-kv 系列支持、CUDA Graph 完整集成、大量 kernel 重构与量化增强、并行框架改进、以及全面更新的文档体系。

---

✨ 核心亮点

1. 💎 SVDQuant W4A4 量化

Cache-DiT v1.5.0 原生集成了完整的 SVDQuant W4A4 量化工作流，这是本次发布最重要的特性。与其依赖第三方库，用户现在可以直接通过 Cache-DiT 的 cache_dit.quantize() / cache_dit.load() API 完成从校准到推理的全链路 W4A4 量化。

📊 PTQ（后训练量化）

支持 svdq_int4_r{rank} 和 svdq_nvfp4_r{rank} 两种量化类型：

• INT4 PTQ（≥sm80）：通过 calibrate_fn 收集激活统计 → SVD 低秩分解 → INT4 打包 → 运行时 W4A4 GEMM。支持三种校准精度：low（推荐默认，~18x 加速）、medium、high。序列化到 {quant_type}.safetensors + quant_config.json，通过 cache_dit.load() 一键恢复。
• NVFP4 PTQ（≥sm120，Blackwell）：专为 RTX 5090 等 Blackwell GPU 设计，使用 NVFP4 格式打包权重，当前仅支持 runtime_kernel="v1"。

性能数据（FLUX.2-klein-4B，1024×1024，L20）：

阶段	延迟 (s)	显存 (GiB)	Transformer 权重 (GiB)
BF16 baseline	2.13	17.32	7.22
SVDQuant INT4	1.24	12.39	2.28
SVDQuant + compile	1.02	12.39	2.28

• Transformer 权重：~3.2× 压缩（7.22 → 2.28 GiB）
• 端到端延迟：~1.7× 加速（2.13 → 1.24s），compile 叠加后 ~2.1× 加速（2.13 → 1.02s）
• PSNR 保持 29+ dB，视觉质量几乎无损

NVFP4 性能（RTX 5090，FLUX.2-klein-4B，1024×1024）：

阶段	延迟 (s)	加速比	显存 (GiB)
BF16 baseline	0.97	1.00×	17.32
NVFP4 PTQ	0.58	1.69×	12.50
NVFP4 + compile	0.47	2.05×	12.50

⚡ DQ（动态量化）

无需校准数据的零样本量化，类型后缀 _dq（如 svdq_int4_r128_dq）：

• identity（默认）：直接对原始权重矩阵做 SVD 低秩分解，无需校准、无需序列化
• weight / weight_inv：仅基于权重统计量的启发式平滑策略（实验性）
• few_shot：运行时收集少量前向的激活统计后实时量化，支持 7 种松弛策略（auto/stable_auto/power/log/rank/top/fixed），可配置 few_shot_steps、few_shot_relax_factor、few_shot_relax_top_ratio、few_shot_relax_strategy。支持 few_shot_auto_compile 在量化完成后自动触发 torch.compile

DQ 性能（FLUX.2-klein-4B，1024×1024，identity smooth，rank=128）：1.28s，PSNR 28.71 dB。

🔧 SVDQ Converter CLI

新增 cache-dit-convert 命令行工具，一键将预训练模型转换为 SVDQ W4A4 格式：

cache-dit-convert --model-path black-forest-labs/FLUX.2-klein-4B \
  --save-dir ./FLUX.2-klein-4B-svdq \
  --quant-type svdq-int4-r128-dq

支持 INT4/NVFP4 DQ 转换、自定义 smooth 策略、多种精度选项。生成 {quant_type}.safetensors + quant_config.json，可通过 cache_dit.load() 直接加载。

🔀 Fused MLP

新增 fused_gelu_mlp / fused_gelu_proj pass（通过 svdq_kwargs["fused_mlp"]=True 启用），将第一个 GEMM + GELU 激活融合为单 kernel，降低 kernel launch 开销和中间激活显存占用。

🔗 与并行兼容（Cache-DiT 独家特色）

SVDQuant 在 Cache-DiT 中完整支持 Context Parallelism（Ulysses / Ring / USP）和 Tensor Parallelism（PyTorch DTensor）。用户可以在分布式推理场景下叠加使用量化 + 并行，实现极致的显存压缩和吞吐提升。SVDQW4A4ShardLinear（dtensor.py）提供原生 TP 分片支持。这是 Cache-DiT 中 SVDQuant 区别于其他 W4A4 实现的差异化能力。

⚙️ 量化配置增强

• 区域量化（regional_quantize=True + repeated_blocks）：仅量化 transformer 的重复块，保持嵌入层等敏感层全精度
• 混合精度计划（precision_plan）：按层名模式为不同子层指定不同量化类型（如 attn.to_q 用 float8_per_tensor、attn.to_k 用 float8_weight_only）
• FP8 Per-Tensor Fallback（per_tensor_fallback=True，默认开启）：在 TP 场景下，不支持 per-row 量化的层自动回退到 per-tensor，消除跳过警告，提升覆盖率（144 → 144，0 skip）
• TorchAO 后端重构：更清晰的 QuantizeBackend 枚举（AUTO / TORCHAO / CACHE_DIT / NONE）
• 量化 API 重构：废弃旧版 kwargs，统一为 QuantizeConfig + svdq_kwargs

📦 cache-dit-cu13 预编译 Wheel

为 CUDA 13 用户提供预编译的 SVDQuant wheel：pip install cache-dit-cu13，免去从源码编译 SVDQ kernel 的麻烦。

---

2. 💾 Bucket-style Layerwise CPU Offload

Cache-DiT v1.5.0 引入了全新的 bucket 式逐层卸载机制，解决了传统逐层 offload "每层等传输"的低效问题。

核心设计：

• Bucket Pipeline：将目标模块切分为连续小桶，当前桶执行时异步预取下个桶，实现计算-通信重叠
• 双独立 Copy Stream 池：onload（H2D）和 offload（D2H）各自拥有独立 CUDA stream 池
• Persistent Bins：将常驻预算均匀分布到目标序列上，避免热权重集中在 prefix
• 灵活的资源控制：transfer_buckets（预取深度）、persistent_buckets（常驻桶数）、persistent_bins（常驻分布桶数）、prefetch_limit（保守预取限制）、max_copy_streams（并发拷贝流数）、max_inflight_prefetch_bytes（预取字节预算）

性能数据（FLUX.1-dev，L20）：

配置	显存	延迟
无 offload	~38 GiB	23.4s
原生 diffusers offload	~25 GiB	56s
原生 diffusers sequential	~1 GiB	335s
Layerwise（transfer=4, persistent=32, bins=4）	~16 GiB	24.6s

仅 1.2s 的额外延迟（vs 无 offload 23.4s），即可将显存

[!IMPORTANT]
✂ PR body was truncated to here.

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • At any time (no schedule defined)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.