[None][feat] Support Inkling-NVFP4 model by KleinBlueC · Pull Request #17062 · NVIDIA/TensorRT-LLM

KleinBlueC · 2026-07-30T09:35:22Z

Dev Engineer Review

Adds PyTorch support for InklingForConditionalGeneration with hybrid attention, relative-position bias, short-convolution state, sigmoid-gated MoE routing, NVFP4 decoding, and BF16 vision/audio towers.
Adds Triton attention kernels, multimodal preprocessing, checkpoint mapping, KVCacheManagerV2, CUDA-graph handling, reasoning parsing, and evaluation integration.
Adds configuration registration, EOS fallback handling, quantization-value normalization, and public model exports.
Adds documentation and Blackwell accuracy references for GSM8K, MMLU, and MMMU.
Review focus:
- Validate Triton kernel correctness, cache page handling, relative-bias shapes, sliding-window masking, and decode output reuse.
- Validate request-scoped convolution state cleanup and CUDA-graph preparation across warmup, capture, replay, and steady-state execution.
- Validate NVFP4 expert weight mapping, tensor slicing, padded LM-head trimming, and complete checkpoint accounting.
- Check multimodal placeholder expansion, image/audio/video geometry, dtype handling, and unsupported hash-prefix caching behavior.
- Check API consistency, error handling, performance, and configuration values against CODING_GUIDELINES.md.
- Investigate reported failures in L0_MergeRequest_PR pipelines 51386, 51419, 51449, 51605, 51668, and 51832 before another CI run.

QA Engineer Review

Added model tests:
- TestInkling_NVFP4.test_nvfp4 in text and multimodal accuracy suites.
- Inkling registration, geometry, layer classification, checkpoint patterns, weight accounting, and tensor-shape tests.
- Vision, audio, video, processor, placeholder, token-ID, and cache-prefix tests.
Added parser tests for typed content, reasoning, tool invocation, control-token boundaries, streaming, registration, and special-token handling.
Added sampling-parameter tests for EOS precedence, EOS lists, explicit stops, and missing EOS values.
Added LM evaluation tests for post-processing resolution, token-limit preservation, special-token retention, and Inkling/MMMU extraction.
Added CI and QA entries in:
- tests/integration/test_lists/qa/llm_function_core.txt
- tests/integration/test_lists/test-db/l0_b200.yml
The added accuracy tests are covered in both QA and post-merge test lists.
Verdict: needs follow-up because the reported L0_MergeRequest_PR pipelines failed and CBTS coverage data is unavailable.

Description

Adds PyTorch-backend support for Inkling (thinkingmachines/Inkling-NVFP4), a
RoPE-free hybrid-attention MoE reasoning model with vision and audio towers.
Registered as InklingForConditionalGeneration.

Architecture

No RoPE — position comes from a learned relative-position bias added
pre-softmax, per (query token, head, relative distance).
Hybrid attention — 55 local sliding-window layers (window 512, 16 KV heads)
interleaved with 11 global full-causal layers (8 KV heads).
Four causal short convolutions per layer — two inside attention (K, V) and two
on the residual stream, carrying per-request state across decode steps.
Sigmoid-gated MoE — top-6 of 256 routed experts with an additive selection
bias, then a log-sigmoid renorm spanning the routed and two shared-expert logits.
muP + unpadded vocab — hidden states divided by logits_mup_width_multiplier
before the head; logits sliced from 201024 down to 200058.
Multimodal — hMLP vision tower and dMel audio codebook tower, both BF16, each
fusing one row per patch / per frame into the text embedding stream. Video is
multi-frame images; there is no separate video encoder.

NVFP4 covers the text decoder and routed experts (layers 3–65); layer-2 experts,
attention, shared experts, and both towers stay BF16 per the checkpoint's
hf_quant_config.json.

Implementation

Area	Notes
`modeling_inkling.py`	Decoder stack, MoE, short convolutions, multimodal wrapper
`attention_backend/inkling_triton.py`	Paged prefill + decode Triton kernels applying the relative bias as a `score_mod`. No existing fused backend exposes such a hook (context FMHA is disabled for `kRELATIVE`, trtllm-gen rejects a relative bias, FlashInfer has no additive per-token bias). `rel_logits` is static-shape, so the decode kernel is CUDA-graph capturable
`modeling_inkling_multimodal.py`	Vision tower, audio tower, video frame sampling, and the shared input processor that expands one `<image>` placeholder per patch and one `<audio>` per dMel frame. Pure numpy + torch preprocessing
`configs/inkling.py`	Config classes rebuilt from the checkpoint's nested dicts (transformers does not ship Inkling), plus the per-layer classification helpers used by both the model and KV-cache sizing
`checkpoints/hf/inkling_weight_mapper.py`	Checkpoint accounting (every key is consumed-text / deferred / unaccounted) and load-path remapping, including de-interleaving the gate/up projections
`pyexecutor/*`	Registers the short-conv state pool under a new `CONV_STATE_MANAGER`. Pool rows and attention decode metadata are published into stable CUDA buffers eagerly from input prep, so the captured decode forward does no host→device copy and each replay reads the current batch (same stable-pointer pattern as `Mamba2Metadata`)

KV cache: the per-layer KV-head split (local 16 / global 8) structurally requires
KVCacheManagerV2 — V1's unified pool would coerce it to one value and mis-size the
per-layer KV bytes, a correctness bug. The model defaults to V2 and the incompatible
path raises rather than silently downgrading.

Serving / eval plumbing (small, each needed end to end):

--reasoning_parser inkling for Inkling's typed-content blocks, with streaming.
needs_raw_special_tokens so a delimiter-based reasoning parser actually sees its
markers (previously only the tool-parser path preserved them).
end_id falls back to the config's eos_token_id — checkpoints whose terminator
lives only in config.json otherwise never stop and run to max_tokens.
hf_quant_config.json may spell "no quantization" as the string "none", which
previously reached QuantAlgo("none") and raised.
Warmup dummy ids are bounded to the head vocab; a vocab padded above the output
vocab otherwise fails KV-cache estimation with "Token ID out of range".
--post_process_fn inkling / inkling_mmmu for trtllm-eval.

Accuracy

Measured on the complete datasets, TRT-LLM and SGLang side by side at TP=4 with
CUDA graph and the overlap scheduler on, batch 8, driven by the same client so
prompt rendering and scoring are shared code:

Task	TRT-LLM	SGLang	Δ
GSM8K (1319)	96.13 (1268/1319)	96.59 (1274/1319)	−0.45 pt
MMLU (14042, 5-shot)	85.35 (11985/14042)	85.48 (12003/14042)	−0.13 pt
MMMU (858)	79.72 (684/858)	78.32 (672/858)	+1.40 pt

Not supported in this release

MTP / speculative decoding (the checkpoint ships next-N draft weights; nothing
builds or loads them), LoRA, function calling, constrained/guided decoding, EPD
disaggregated serving, and multimodal-hash prefix caching (refused loudly per
modality, so multimodal requests still run — just uncached).

One workaround worth flagging for review: every Inkling all-reduce is rebuilt with
ONESHOT after construction. Under CUDA-graph capture a symmetric all-reduce
corrupts the run when its send buffer is unregistered while its recv buffer is a
registered NCCL window at a 12288 B message — which Inkling hits exactly (hidden
6144, bf16, one decode token), sending the first global-attention layer non-finite.
The all-reduces involved are built by generic modules (attention o_proj, MoE
down_proj), so pinning the strategy afterwards keeps the mitigation model-local.

Test Coverage

Accuracy (integration) — added to l0_b200 and llm_function_core:

accuracy/test_llm_api_pytorch.py::TestInkling_NVFP4::test_nvfp4 — GSM8K + MMLU
on the text decoder.
accuracy/test_llm_api_pytorch_multimodal.py::TestInkling_NVFP4::test_nvfp4 —
MMMU on the vision path.

Unit — CPU-only, no checkpoint / GPU / network needed, all well under a second:

unittest/_torch/modeling/test_modeling_inkling.py — config parsing,
registration, per-layer classification; plus checkpoint-gated weight accounting
and tensor-shape checks that read only the safetensors index and skip cleanly
when the checkpoint is absent.
unittest/_torch/modeling/test_modeling_inkling_multimodal.py — the three media
paths and the input processor on synthetic configs: hMLP scale plan and module
tree, the fold's value preservation, dMel preprocessing, the audio codebook
forward against a reference, frame sampling, and the fail-loud placeholder
contract.
unittest/llmapi/test_reasoning_parser.py — full-parse and streaming equivalence
for the Inkling parser, including control tokens split across delta boundaries.
unittest/llmapi/test_sampling_params.py — the end_id config fallback.
unittest/others/test_lm_eval.py — the offline post-processing hook.

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Add the Inkling NVFP4 model to the _torch stack: modeling_inkling.py, a Triton score_mod attention backend (SWA + global + relative-position bias) over KVCacheManagerV2, HF NVFP4 weight mapper/configs, trtllm-gen blockScaleMoe runner with sink-renorm routing, reasoning-parser/effort rendering, lm_eval post-processing, and the inkling_* unittest suite. Progress (working snapshot): - Component validation passes in isolation on the TP=4 NVFP4 / trtllm-gen MoE stack: weight load & accounting, attention source-activation replay, MoE replay, and full-model source-logit replay. - Baseline (cuda_graph=off, overlap=off) accuracy vs the SGLang reference: GSM8K 0.916 vs 0.972 (-5.6pt), full MMLU 82.22 vs 85.66 (-3.44pt). The gap is dominated by runaway / non-terminating generation on hard prompts: 76% of GSM8K errors are 7k-8.6k-token spirals where the model reaches the answer but never emits EOS, while SGLang commits. Per-layer localization is exhausted; the residual is a diffuse fp4 / kernel-family divergence (bf16 Triton attention + trtllm-gen fp4 MoE vs SGLang flashinfer), not a single fixable layer bug. - Enabled (cuda_graph=on) is blocked by a decode collapse (B2) localized to the global-attention block under CUDA-graph capture/replay at TP=4: h_attn goes non-finite at the first global-attention layer once decode crosses a KV-page boundary; a reduced-model TP=2 harness reproduces it. Next: decode-side termination fix for the baseline runaway (finish_reason-based detection + EOS/stop handling), and resolve B2 in the global-attention-under- graph path. Excludes build artifacts (libtensorrt_llm.so), locks, and the ext/ submodule. Signed-off-by: kleinc <kleinc@nvidia.com>

…(6*448)) Root cause of the baseline NVFP4 accuracy gap vs SGLang. The Inkling checkpoint ships routed-expert activation calibration as a RAW `.input_amax`, but the fused-MoE loader / trtllm `fp4_quantize` expect the ModelOpt per-tensor `input_scale = amax / (E2M1_MAX * E4M3_MAX) = amax / (6*448)`. The mapper renamed `.input_amax` -> `input_scale` WITHOUT the conversion, making the activation global scale 2688x too small: every routed expert's fp4 output was ~0.62 rel_rms from the bf16 ground truth (7.6x SGLang's 0.082). Mirrors sglang inkling.py:1222 / inkling_common/dense_mlp.py:497. Positional bisection confirmed the weight/block-scale/gate-up-interleave layout was already element-wise correct (24/24 L3 experts, 18.8M elems, 0 diverged) -- the defect was ONLY the activation input scale, shared by both TRT MoE backends. Fix: `inkling_weight_mapper._map_expert` divides `.input_amax` by (6*448). Result (baseline: cuda_graph=off, overlap=off, TP=4, CUTLASS MoE, fp4 fix active): - L3 routed-expert vs bf16 truth: rel_rms 0.624 -> 0.081 (== SGLang 0.082) - GSM8K paired 5x100: TRT 0.916 -> 0.968 vs SGLang 0.974; mean_delta -0.056 -> -0.006 (Gate 2 PASS, within +/-0.02); runaway canary passes; collapse 0 - MMLU paired 6x100: TRT ~0.822 -> 0.862 vs SGLang 0.872; gap -3.44pt -> -1.0pt (Gate 3 accuracy PASS); B-bias error-is-B 65.9% -> 40%, TRT-B 33.2% -> 29.8% Residual (not this fix): TRT (CUTLASS) vs SGLang (flashinfer) fp4 kernel-family near-tie noise -- accuracy-neutral, not bit-reproducible at batch>1. The strict within-2pp-of-gold MMLU B-bias criterion is model-inherent (SGLang itself over-picks B) and is left to human adjudication, not a TRT defect. Also adds env-gated per-layer/per-module dump instrumentation (modeling_inkling.py dump_sink; inkling_perlayer_localize_test.py) and the trtllm-gen MoE backend path used to localize the bug. Signed-off-by: kleinc <kleinc@nvidia.com>

…ring graph capture Root cause of the Inkling TP=4 enabled-runtime (cuda_graph=on + overlap) decode collapse ("B2"): the AutoTuner-selected all-reduce (`tunable_allreduce`, AUTO strategy) is not CUDA-graph-capture-safe. Frozen into a decode graph, the tuned tactic produces a non-finite result on replay, so decode goes NaN from the first global-attention layer and collapses to a token-0 repeat ("Paris!!!!"). Eager is finite at identical metadata; the fault is baked into the captured graph. Localized by single-variable determinism isolation (autotuner ON -> collapse, autotuner OFF -> clean), NOT by op-fingerprints -- any in-graph probe shifts the graph memory pool / tactic selection and suppresses the bug (a Heisenbug). Fix (`tensorrt_llm/_torch/distributed/ops.py`, AllReduce.forward): during CUDA-graph capture (`torch.cuda.is_current_stream_capturing()`), skip `tunable_allreduce` and fall back to the static, graph-safe `all_reduce_op(AUTO)`. Warm-up / eager (not capturing) still autotune, so steady-state performance is unchanged. This is a mainline TRT-LLM robustness fix (any TP model that captures an AUTO all-reduce benefits), not Inkling-specific. Confirmed: enabled generation_parity is BIT-IDENTICAL to the baseline (cuda_graph=off) -- tf_mismatch/neartie/confident=16/11/5, freerun_collapse=0, identical logit_checksum; enabled 5x100 GSM8K (0.966 vs SGLang 0.974) and MMLU (0.875 vs 0.872) within +/-0.02, zero errors, no regression vs the accepted CUTLASS baseline. Also includes env-gated per-layer/per-op B2 localization instrumentation (`modeling_inkling.py` dump_sink / INKLING_FP*; `inkling_fp_localize_test.py`), zero-cost when the INKLING_FP* env is unset. Signed-off-by: kleinc <kleinc@nvidia.com>

…MLP tower, image fusion (WIP) WIP -- Stage-1 progress snapshot on the Inkling NVFP4 multimodal tower, on top of the accepted text tower. Not finished: the vision path is verified clean but the MMMU Accounting gap is still open (decode-side), and audio / MTP are still deferred. Committed to record current progress, not as a complete feature. Text decode/attention/MoE paths are untouched. Model / config: - configs/inkling.py: add image_token_id (200054, the in-vocab chat-template <|unused_200054|>) and audio_token_id. The SGLang-internal -101 sentinel is rejected by TensorRT-LLM's executor token-id validation; the two ids are interchangeable for parity since both are overwritten by vision embeddings. - modeling_inkling_vision.py (new): hMLP vision tower InklingVisionModel and InklingInputProcessor, which expands the <image> placeholder to one token per vision patch and attaches vision_patches_bthwc features. - modeling_inkling.py: InklingForConditionalGeneration registers the input processor, builds the vision tower as a replicated bf16 submodule, and fuses per-patch embeddings into the text stream via fuse_input_embeds with explicit text/mm indices (OOV-safe). Fixes the "image not visible" hallucination: the fused stream must NOT be re-normed, since SGLang scatters raw vision rows in after embed_norm; the extra RMSNorm corrupted the image rows. Tests / diagnostics (tests/unittest/_torch/modeling/, 25 new files): MMMU harness alignment against the SGLang scorer, input-processor and vision-tower unit checks, image e2e / fusion / logit-replay / generation-parity drivers, and the localization probes used to isolate the vision-vs-decode split (vision verified bitwise-clean; the residual Accounting gap is decode-side). These require GPU + the NVFP4 checkpoint (TP=4) and were not run for this commit. Signed-off-by: kleinc <kleinc@nvidia.com>

…s_token SamplingParams._setup() looked for the end-of-generation token in only two places: tokenizer.eos_token_id, then generation_config.eos_token_id. When a checkpoint provides neither, end_id stayed None, nothing could terminate a request, and every generation silently ran to max_tokens. That combination is reachable. Multi-part chat formats have no single terminator -- a message end, an end-of-sampling marker and a document separator are distinct tokens -- so such checkpoints register their control tokens under extra_special_tokens / additional_special_tokens, neither of which populates tokenizer.eos_token_id, and declare the real stop token as eos_token_id in config.json. Those checkpoints also tend to ship no generation_config.json. _setup() already received hf_model_config but never consulted it. Observed on a checkpoint of that shape: responses ran to the token limit while emitting the configured eos_token_id up to 441 times in a single response. Fall back to hf_model_config.eos_token_id when the first two sources yield nothing. A list value sets end_id from its first entry and appends the rest to stop_token_ids, mirroring the existing generation_config path. Priority is otherwise unchanged: an explicit SamplingParams(end_id=...) still wins and tokenizer.eos_token_id still takes precedence over config.json, so models that already resolved an end_id see no behavioural change. Warn when all three sources come up empty. Generating to the cap on every request with no diagnostic is the part that makes this expensive to find. Add unit tests for tokenizer priority, the config fallback, the list form, an explicit end_id not being overridden, the all-empty case, and a missing hf_model_config. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

Adds the audio and video modalities alongside the existing vision path. Audio: * InklingAudioPreprocessor -- dMel feature extraction (mel basis, hz<->mel, per-frame bin quantization) producing int32 [N, 80] bins, one audio token per frame. * InklingAudioModel -- codebook encoder (bin m occupies codebook rows [m*V, (m+1)*V), summed over bins) plus optional final norm, bf16, loaded strictly from the real checkpoint's audio tensors. * Placeholder expansion and fail-loud count checks in the input processor. Video: * sample_video_frames / sample_video_as_images / DecodedVideo -- frame sampling ported to match SGLang's sample_video_frames semantics. * The <image>-per-frame path: every sampled frame becomes its own image span through the existing vision tower. Audio and video land together because both wire into the same InklingInputProcessor.assemble dispatch; splitting them would leave an intermediate commit whose processor references helpers that do not exist yet. Tests (all GPU-verified on TP=4): * inkling_audio_tower_test.py -- 10 passed, incl. real-weight CUDA forward (AUDIO_TOWER_CUDA_OK, out=(n_frames, 6144) bf16 finite) and a reference-math allclose(atol=1e-5) check of the codebook sum. * inkling_video_utils_test.py -- 12 passed, incl. a port of SGLang's test_video_utils.py::test_sample_video_frames_lengths (same 4 cases and the same expected frame indices) and a real-weight multi-frame CUDA forward (VIDEO_TOWER_CUDA_OK, out=(total_patches, 6144)). * inkling_audio_e2e_test.py / inkling_video_e2e_test.py -- strict TP=4 end-to-end smokes: every prompt must be finite, non-empty and non-collapsed. Green both baseline (5/5) and with cuda_graph+overlap enabled (5/5). Scope note: these prove the modalities run and are shape/dtype/finiteness correct; they are not a cross-stack numerical parity check against SGLang. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

`logits_rows` is a view of the same slice that is written back, so the assignment self-overlaps and torch rejects it with "... refer to a single memory location. Please clone()...". The processors edit in place, so the write-back is redundant anyway; cloning the source makes it overlap-safe. Only reached for requests carrying a py_logits_post_processor, so normal requests are unaffected. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

… parity runs evaluate/interface.py gains summarize_generation_stats/log_generation_stats, and lm_eval emits a greppable GEN_STATS marker per batch. Without it a runaway / no-EOS regression hides behind a parseable answer buried in a wall of repeated text -- the failure mode that cost several bring-up iterations. The MMMU harness/runner changes carry the sharded union runs used for the TRT-vs-SGLang comparison (shard plan, incremental atomic per-item writes so a wall-killed shard keeps everything it scored, cap/max_seq plumbing), plus unit coverage of the answer parser. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

Adds streaming-vs-batch equivalence over arbitrary split points, tool-call and repetition segmentation, and end-tokens split across deltas. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

Standalone probes used to localize the TRT-vs-SGLang vision divergence: prefill logits, transformers-reference decode, termination behaviour, and a per-layer activation dump. Diagnostics only -- not part of any test suite. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

Two in-progress pieces, neither reached by a runtime path yet. MTP static tier: InklingMTPConfig carries the checkpoint's num_nextn_predict_layers / chain_hidden_post_norm / local_layer_ids plus the per-depth banded-attention geometry injected from the text tower, so draft depths in local_layer_ids get SWA head geometry and the rest stay global. The weight mapper gains inkling_expected_mtp_keys() and accounts model.mtp.* as consumed rather than deferred when an mtp_config is supplied; the default mtp_config=None leaves the text-tower accounting byte-identical. Unit coverage for config parse, weight accounting, the BF16/unquantized requirement, and per-depth banding against the checkpoint shapes. MMMU harness: INKLING_MMMU_TEXT_ONLY reruns the same items as pure text (image placeholder stripped, no image attached) so the shared decoder is exercised at the identical bs / cap / overlap regime without the vision path. Default-off — the vision scoring path is unchanged when unset. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…del code The cuda-graph decode collapse is a defect in symmetric all-reduce: a captured NCCL_SYMMETRIC reduce whose send buffer is unregistered while its recv buffer is a registered NCCL window corrupts the run at a 12288 B message. Inkling hits that size exactly -- hidden 6144, bf16, one decode token -- so the first global-attention layer goes non-finite and decode collapses to a repeated token 0. Revert the shared-code mitigation in distributed/ops.py: that file is now byte-identical to its pre-Inkling state, so no other model's all-reduce path changes on our account. Mitigate in modeling_inkling.py instead. The all-reduces that trigger this are built by generic modules -- attention o_proj, MoE down_proj -- so the strategy cannot be passed at construction without editing shared code; rebuilding each AllReduce after super().__init__() keeps the mitigation model-local. Each rebuilt instance carries the module's own mapping and dtype over, so strategy is the only delta. Pinning ONESHOT also drops the window requirement, since AllReduce only takes an NCCL window under NCCL_SYMMETRIC/NCCL/AUTO -- two of the five trigger conditions go away, not just one. Active by default; INKLING_ALLREDUCE_STRATEGY=AUTO restores stock behaviour, and the defect with it, for A/B runs. Measured on job 5728192, alternating arms in one job against one binary: default 0/3 collapse, AUTO 3/3, 331 modules swapped. Cost: symmetric is disabled on every Inkling all-reduce, eager included, and roughly a third of captured decode all-reduces pick it today. The performance impact is unmeasured and should be measured before this is treated as final. This is containment, not a root-cause fix -- the defect remains for any other model that meets all five conditions. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

Removes the bring-up's debug scaffolding and completes the deliverables the Inkling NVFP4 change was missing. No model behaviour changes. Deleted 41 debug/localization test files (divergence probes, dump/isolate helpers, teacher-forcing and graph-capture localizers) that existed to find the bring-up's defects and have no role now that it works. Nothing imports them: every deleted basename was grepped across the tree, zero references remain. The two environment variables the model still reads are the ones worth keeping — INKLING_ALLREDUCE_STRATEGY (the escape hatch for the ONESHOT all-reduce mitigation) and INKLING_MOE_BACKEND (the trtllm-gen MoE kernel select) — and both are documented; the 19 debug-only INKLING_* knobs are gone. Completed the deliverables: - docs/source/models/supported-models.md — architecture row, multimodal feature-matrix row, and footnote [^14] covering modality coverage, the unsupported set (MTP, LoRA, function calling, constrained decoding, EPD, mm-hash caching), and the all-reduce mitigation plus its escape hatch. - TestInkling_NVFP4::test_nvfp4 added to the shared multimodal accuracy file rather than a standalone per-model test, with a sourced MMMU reference. - Registered that id in test-db/l0_b200.yml and qa/llm_function_core.txt. - Dropped the orphaned inkling_vision_tower_artifact.json; the surviving regression test consumes the generated artifact instead. Static checks: git diff --check clean, every modified Python file compiles, no stale references to the deleted files. Runtime coverage as of the last completed suite: unit tiers green (vision 43, audio 10, video 12, text 31, collect 95 with zero residual debug files) and GSM8K cg0ov0 parity at delta=0.0. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

The MTP / next-N draft work never reached a runtime path: nothing builds the draft layers and nothing consumes them. Only the config plumbing and the load-accounting existed, so remove them rather than ship dead code. - configs/inkling.py: drop InklingMTPConfig, InklingConfig.has_mtp and _as_mtp_config. mtp_config goes back to a plain retained blob, so the checkpoint still round-trips. - inkling_weight_mapper.py: drop inkling_expected_mtp_keys and the consumed_mtp bucket; inkling_account_checkpoint loses its mtp_config parameter. - test_modeling_inkling.py: drop the four Stage-9 MTP tests. - supported-models.md: the footnote no longer claims the draft weights are weight-accounted. Checkpoint accounting is unaffected: "model.mtp." stays in INKLING_DEFERRED_PREFIXES, so the draft weights are classified as deferred exactly like the audio and vision blocks, and `unaccounted` stays empty. Verified: every touched file compiles, and no reference to any removed symbol remains anywhere in the tree. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…ences MMMU already had a reference entry; the two text benchmarks the bring-up actually measured did not, so the numbers lived only in run logs. Both references are the cached SGLang NVFP4 measurement under the same harness, matching how the MMMU entry was sourced: - GSM8K 95.53 (full set, 0.9553). TRT-Inkling was validated against it with the paired 5x100 protocol: flexible-parse mean 0.968, and all four cuda-graph x overlap-scheduler corners scored 0.98-0.99 on the 100 paired items where SGLang scored 0.99. - MMLU 85.66 (full Hendrycks, 14042 samples, weighted_accuracy 85.6573). The TRT side was measured with the 5-seed text-regression protocol on the harness' 114-item subset (83.33 / 84.21 / 85.09 / 85.96 / 87.72, mean ~85.3). The comment says so explicitly: it tracks the reference, but no full-set TRT-LLM MMLU run has been recorded yet. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…that matters The bring-up left 22 Inkling files under tests/unittest/_torch/modeling. Most were per-defect regressions or SGLang-alignment scaffolding whose outcome is now covered end-to-end by the MMMU/GSM8K accuracy tests. Keep one test per thing that nothing else covers, drop the rest. Kept: - test_modeling_inkling.py config parse, layer classification, weight accounting over the real checkpoint - inkling_vision_tower_test.py vision tower - inkling_audio_tower_test.py audio tower (no accuracy benchmark covers it) - inkling_video_utils_test.py video utils (no accuracy benchmark covers it) - inkling_input_processor_test.py multimodal input processor - inkling_moe_backend_select_test.py the INKLING_MOE_BACKEND kernel select inkling_mmmu_real_align_test.py was doing double duty: the vision-tower and input-processor tests import its MMMU item fetch/cache and its importlib loader. Split that half out as inkling_mmmu_fixtures.py (a fixture module, no tests) and drop the SGLang-alignment machinery with the rest. Deleted (16): the mmmu align/harness/run/parser set, image_prompts, the three per-modality e2e smokes, image_fusion, image_norm_fix, attn_decode_meta, gate_up_deinterleave, kv_manager_v2, generation_parity and source_logit_replay. Every kept file compiles and no reference to a deleted module remains in the tree. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

The multimodal file already carried TestInkling_NVFP4::test_nvfp4 for MMMU, so the vision path had an integration test but the shared text decoder -- what GSM8K and MMLU actually exercise -- had none. Adds TestInkling_NVFP4 to test_llm_api_pytorch.py running both text benchmarks against the references recorded earlier. It mirrors the multimodal class: NVFP4 assert, 16384-token budget for the long chain of thought, and extract_inkling_content as the post-processor so the <|content_thinking|> channel is dropped and only the visible answer is scored. Registered the new id in both lists that already carry the multimodal one: test-db/l0_b200.yml and qa/llm_function_core.txt. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…keep CUTLASS The Inkling routed experts now run only the default CUTLASS MoE backend. The trtllm-gen blockScaleMoe path was an opt-in experiment behind INKLING_MOE_BACKEND=TRTLLM, never the default, and it needed a dedicated routing enum plumbed all the way into the CUDA runner to work. Restored to their pre-bring-up state (the additions were Inkling-only, so these files are now byte-identical to 44f0521): - cpp/.../trtllmGenKernels/blockScaleMoe/runner.h the InklingSinkRenorm = 9 enum value and its name case - cpp/.../trtllmGenKernels/blockScaleMoe/runner.cu the precomputed-routing dispatch branch - _torch/modules/fused_moe/routing.py the matching Python enum value and its autotuner-dummy mapping Removed from the model: - _inkling_trtllm_moe_backend / _moe_config_with_trtllm_backend and the frozen-config copy they needed to retarget moe_backend - InklingMoeRoutingMethod._trtllm_backend, so routing_method_type is always Unspecified and requires_separated_routing goes back to the default - the per-layer INKLING_MOE_SELECT backend-introspection log Also deletes inkling_moe_backend_select_test.py, which existed only to cover that knob, and the stale comment that pointed at it as the fix for the fused combine's cross-row non-determinism. INKLING_ALLREDUCE_STRATEGY is now the only environment variable the model reads. Everything compiles and git diff --check is clean. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

The MMLU comment added in 5f28628 claimed no full-set TRT-LLM run had been recorded. That is wrong. One exists: 2026-07-20, all 14042 Hendrycks samples, weighted_accuracy 82.2176 -- 3.44 points below the 85.66 reference, and it failed the bring-up's 2-point gate. The gap was later traced to an fp4 expert-GEMM family difference (CUTLASS/trtllm-gen vs SGLang's flashinfer) plus non-terminating generation on hard prompts. What is true is narrower: the full set has not been re-measured since those fixes. Post-fix MMLU evidence is subset-only -- a 570-item stratified canary at 84.21 (-1.45, inside the gate) and a 5-seed 114-item regression averaging ~85.3. The comment now says exactly that, including that meeting 85.66 at full scale is unverified. GSM8K's comment was not wrong but was too vague about coverage. TRT-LLM has never been measured on the full 1319-item set: the evidence is the paired 5x100 protocol (flexible mean 0.968), the four cuda-graph x overlap corners at 0.98-0.99 on 100 paired items, and one early full-set attempt that completed only its first 120-item chunk (0.925 vs 0.975). Spelled out. Both files still record the SGLang reference as the accuracy value, so TestInkling_NVFP4::test_nvfp4 grades against a bar the model has not been shown to clear at full scale. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…timodal The module already carried the image, audio and video paths, so the "vision" name no longer described it. Rename it and reorganize the contents into four labelled sections -- vision tower, audio tower, video tower, and the shared multimodal input processor -- so each modality is easy to locate. Behavior-preserving. Along the way: * factor the duplicated tower weight-loading into ``_load_tower_weights`` and the duplicated RMSNorm into a single shared ``InklingRMSNorm`` (was ``InklingVisionRMSNorm``, used by both towers); * factor the input processor's repeated media-list coercion and its placeholder/feature-row count checks into small helpers, dropping a tautological per-item check (``num_tokens`` is built from ``num_patches``); * trim the docstrings to what the code needs, dropping the development-time stage/goal references and reference-implementation file paths. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

Bring-up leftovers that do not belong in the production model: * cuda_graph_runner: drop the "[cuda-graph] CAPTURED/REPLAYED" evidence logging and its once-only gate flag. This was pure instrumentation added to prove the runtime really captured and replayed a graph, and it sat in shared (non-Inkling) code. * modeling_inkling: drop the INKLING_ALLREDUCE_STRATEGY environment knob. The ONESHOT pin is a correctness mitigation, not a tuning option, so it is applied unconditionally instead of via an A/B toggle (and the log line that reported it is gone). * modeling_inkling: drop the explicit-window short-conv decode path (InklingShortConv.forward_decode, the decoder layer's third branch, and the attention's conv_states/return_conv_state arguments). Only the bring-up replay harness ever passed those; the runtime always drives the short convs through the per-request state pool. * modeling_inkling: drop the attention's decode_seq_lens / decode_page_table / skip_kv_write arguments. The runtime publishes decode metadata into the layer's stable GPU buffers before capture, so the pre-supplied static tensors had no caller left; the eager fallback that builds them from the host block table stays. * modeling_inkling: drop InklingConvStateCache.reset (unused) and merge InklingConvRuntime.from_metadata into build (the split existed only so the replay harness could publish slots itself). Also rewrite the comments and docstrings across the Inkling _torch files to describe the code as it stands: no development stage/goal numbers, job ids, absolute paths into local reference checkouts, or references to the deleted replay harness. The stale "audio / vision / MTP are deferred" module docstring now reflects that only MTP is unimplemented. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

The Inkling unit tests were five files carrying a bring-up harness rather than a test suite: they loaded the SGLang reference from an absolute path in a local checkout, downloaded MMMU rows over the network into a gitignored cache, wrote JSON artifacts, hardcoded a personal /lustre checkpoint path, and shipped __main__ runners that printed comparison tables. None of that can run in CI. Fold everything multimodal into test_modeling_inkling_multimodal.py -- one file covering the vision, audio and video paths plus the shared input processor -- with small synthetic configs and inputs: no checkpoint, no GPU, no network, no SGLang import, no artifacts. 27 tests, well under a second. Slim test_modeling_inkling.py the same way. The config and layer-classification tests now build their config explicitly instead of requiring the checkpoint, so they actually run; the weight-accounting and tensor-shape tests keep the checkpoint (index JSON only, no weights) and resolve it through the standard llm_models_root() with an INKLING_CHECKPOINT override, so they skip cleanly instead of always skipping on a path that only existed on one machine. Removed: inkling_vision_tower_test.py, inkling_input_processor_test.py, inkling_audio_tower_test.py, inkling_video_utils_test.py (folded in) inkling_mmmu_fixtures.py (MMMU downloader + SGLang loader; no longer used) Also drop the .gitignore entries for the deleted caches and artifacts. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…nkling PR Run the repo's pre-commit formatters (isort, yapf, ruff, ruff-format) over the files this PR touches, and fix the eleven ruff-legacy D205/D209 docstring regressions it flags in tensorrt_llm/evaluate/interface.py and tests/unittest/llmapi/test_reasoning_parser.py. Formatting and docstring wording only; no behavior change. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

Two bring-up leftovers outside _torch: * evaluate: drop summarize_generation_stats / log_generation_stats and both call sites (Evaluator.evaluate and LmEvalWrapper). This logged a greppable GEN_STATS line with the finish_reason mix and generated-token distribution so a runaway/no-EOS regression would be visible while bringing the model up. It is observability scaffolding, not part of the eval contract: it never influenced what was scored, nothing consumed the marker, and no test covered it. * docs: drop "The text decoder is also usable standalone (text-only) via the InklingForCausalLM architecture" from the Inkling footnote. It is not true. InklingForCausalLM carries no @register_auto_model, so it is not in MODEL_CLASS_MAPPING and no checkpoint can select it; the config registry has no inkling_text entry either. The class is the base that InklingForConditionalGeneration derives from, and the inkling_text handling in config_utils/_util exists for that nested sub-config, not for standalone loading. The published checkpoint declares InklingForConditionalGeneration and the text-only accuracy test loads it through that same architecture, so registering the class would advertise a path with no checkpoint to exercise it and no test coverage. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

tensorrt-cicd · 2026-08-06T12:14:30Z

PR_Github #64263 [ run ] completed with state SUCCESS. Commit: 991a1db
/LLM/main/L0_MergeRequest_PR pipeline #52168 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Inkling-Small is the same architecture family as Inkling -- RoPE-free hybrid attention, 256-expert MoE, hMLP vision tower -- at 42 layers / hidden 4096 against 66 / 6144. It therefore needs no modeling change: it loads and runs on the existing InklingForConditionalGeneration path unmodified. What was missing was coverage, so a regression on the smaller checkpoint would go unnoticed. Adds the two accuracy classes as subclasses of the Inkling ones, so the evaluation setup (long-CoT max_tokens, typed-content post-processing, TP=4, KV fraction) stays defined in one place and only the checkpoint differs. References are full-set measurements against SGLang on the same checkpoint, TP=4, greedy, CUDA graph and overlap scheduler on: GSM8K 1319 items 95.75 (SGLang 95.98, delta -0.23) MMLU 14042 items 79.45 (SGLang 79.33, delta +0.12) MMMU 857 items 77.95 (SGLang 77.83, delta +0.12) MMMU was scored item-for-item against SGLang over a canonical token stream shared by both stacks; the two arms differ by one item (668 vs 667) with a 48/47 split on the disagreements, and every scored item used its image. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

…ents CUDA graph capture is broken with expert parallelism enabled, so the feature matrices claim more than holds. Move the column to No for both Inkling rows and say in the footnote which configuration works, so the entry can go back to Yes once the expert-parallel case is fixed rather than being rediscovered. Also trims the comments added with the Inkling-Small references and test classes down to what the reader cannot get from the code beside them. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

KleinBlueC · 2026-08-06T15:55:20Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-06T22:29:39Z

PR_Github #64376 [ run ] triggered by Bot. Commit: badd3e4 Link to invocation

Two conflicts, both the same shape: main added Kimi-Linear entries to sorted lists this branch had added Inkling entries to. Kept both sides in order -- the config __init__ import and __all__, and the config_utils import in pyexecutor/_util.py. No behaviour on either side changes; is_inkling and is_kimi_linear are both still resolved and used. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

KleinBlueC · 2026-08-07T01:51:33Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-07T01:57:44Z

PR_Github #64448 [ run ] triggered by Bot. Commit: 198e857 Link to invocation

The docstring said w13_weight is "gate rows first, up rows second", which is the contiguous layout _split_interleaved_gate_up exists to warn against: the tensor is gate/up-INTERLEAVED, and reading it as two halves pairs the wrong channels in every SwiGLU. The code already splits even/odd rows correctly, so only the description was wrong -- but it described the exact mistake the helper guards, which is worth not leaving in place. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

tensorrt-cicd · 2026-08-07T02:00:08Z

PR_Github #64376 [ run ] completed with state ABORTED. Commit: badd3e4

Link to invocation

KleinBlueC · 2026-08-07T02:01:38Z

/bot run --disable-fail-fast

_experts_are_nvfp4 read "not in exclude_modules" as NVFP4. The exclusion list comes from hf_quant_config.json, which the BF16 release does not ship, so its empty list meant "every layer is quantized" instead of "none are". The expected key set then asked for .scale / .scale2 / .input_amax / .original_shape on every routed-expert tensor: 320 keys the checkpoint never carries, on Inkling-Small. Thread a `quantized` flag through the three accounting entry points and let the test derive it from the presence of hf_quant_config.json. Accounting the BF16 checkpoint now closes exactly -- 0 missing, 0 unaccounted, 1048/1048 covered -- against 320 missing before. This is accounting only; the load path already handled the unquantized case (_map_expert maps a sidecar-less tensor straight to w1/w3/w2 .weight, and a null quant_config builds bf16 modules). Verified end to end on this branch: TP=4 generation from hf_data/Inkling-small produces coherent English and Chinese output, which also confirms the gate/up interleaved split is right for BF16 -- a wrong split would mispair every SwiGLU channel and yield garbage. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

tensorrt-cicd · 2026-08-07T02:07:24Z

PR_Github #64451 [ run ] triggered by Bot. Commit: 2024a24 Link to invocation

tensorrt-cicd · 2026-08-07T02:11:57Z

PR_Github #64448 [ run ] completed with state ABORTED. Commit: 198e857

Link to invocation

KleinBlueC · 2026-08-07T02:28:58Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-07T02:35:19Z

PR_Github #64462 [ run ] triggered by Bot. Commit: 2024a24 Link to invocation

tensorrt-cicd · 2026-08-07T02:39:39Z

PR_Github #64451 [ run ] completed with state ABORTED. Commit: 2024a24

Link to invocation

tensorrt-cicd · 2026-08-07T06:38:53Z

PR_Github #64462 [ run ] completed with state FAILURE. Commit: 2024a24
/LLM/main/L0_MergeRequest_PR pipeline #52338 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

KleinBlueC · 2026-08-07T14:21:07Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-08-07T14:27:50Z

PR_Github #64619 [ run ] triggered by Bot. Commit: 7a45602 Link to invocation

tensorrt-cicd · 2026-08-07T18:27:51Z

PR_Github #64619 [ run ] completed with state SUCCESS. Commit: 7a45602
/LLM/main/L0_MergeRequest_PR pipeline #52479 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

KleinBlueC and others added 5 commits July 20, 2026 19:15

github-actions Bot assigned KleinBlueC Jul 30, 2026

KleinBlueC added 5 commits July 30, 2026 08:13

[None][test] Inkling reasoning-parser streaming equivalence coverage

3167aff

Adds streaming-vs-batch equivalence over arbitrary split points, tool-call and repetition segmentation, and end-tokens split across deltas. Signed-off-by: KleinBlueC <102355066+KleinBlueC@users.noreply.github.com>

KleinBlueC force-pushed the inkling-nvfp4-bringup branch 2 times, most recently from dfc574c to f5fcaf1 Compare July 31, 2026 04:36

KleinBlueC added 2 commits July 30, 2026 21:37

KleinBlueC force-pushed the inkling-nvfp4-bringup branch from f5fcaf1 to 35a4a46 Compare July 31, 2026 04:38

KleinBlueC force-pushed the inkling-nvfp4-bringup branch from d1649bb to 8a32f22 Compare August 3, 2026 03:48

KleinBlueC added 11 commits August 2, 2026 20:53

KleinBlueC force-pushed the inkling-nvfp4-bringup branch from 8658713 to c15285c Compare August 3, 2026 06:08

KleinBlueC removed their assignment Aug 6, 2026

WeiHaocheng changed the title ~~[None][feat] Support Inkling-NVFP4 model~~ [None][feat] BREAKING: Support Inkling-NVFP4 model Aug 6, 2026

WeiHaocheng added the api-breaking Accepted LLM API contract change that is backwards-incompatible label Aug 6, 2026

WeiHaocheng assigned KleinBlueC Aug 6, 2026

KleinBlueC added 2 commits August 6, 2026 08:22

WeiHaocheng added api-compatible Accepted LLM API contract change that is backwards-compatible and removed api-breaking Accepted LLM API contract change that is backwards-incompatible labels Aug 7, 2026

WeiHaocheng changed the title ~~[None][feat] BREAKING: Support Inkling-NVFP4 model~~ [None][feat] Support Inkling-NVFP4 model Aug 7, 2026

KleinBlueC mentioned this pull request Aug 7, 2026

[None][fix] Release MoE workspaces before erasing the CUDA graph pool #17403

Draft

1 task

Merge branch 'main' into inkling-nvfp4-bringup

7a45602

Uh oh!

Conversation

KleinBlueC commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev Engineer Review

QA Engineer Review

Description

Architecture

Implementation

Accuracy

Not supported in this release

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

KleinBlueC commented Aug 6, 2026

Uh oh!

tensorrt-cicd commented Aug 6, 2026

Uh oh!

KleinBlueC commented Aug 7, 2026

Uh oh!

tensorrt-cicd commented Aug 7, 2026

Uh oh!

tensorrt-cicd commented Aug 7, 2026

Uh oh!

KleinBlueC commented Aug 7, 2026

Uh oh!

tensorrt-cicd commented Aug 7, 2026

Uh oh!

tensorrt-cicd commented Aug 7, 2026

Uh oh!

KleinBlueC commented Aug 7, 2026

Uh oh!

tensorrt-cicd commented Aug 7, 2026

Uh oh!

tensorrt-cicd commented Aug 7, 2026

Uh oh!

tensorrt-cicd commented Aug 7, 2026

Uh oh!

KleinBlueC commented Aug 7, 2026

Uh oh!

tensorrt-cicd commented Aug 7, 2026

Uh oh!

tensorrt-cicd commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

KleinBlueC commented Jul 30, 2026 •

edited

Loading