Environment
- Windows 11, single RTX 5090 (32GB, sm_120)
- FreeToken v0.1.2+g141c31a8d, torch 2.11 / CUDA 13
Goal
Serve Qwen3-Coder-30B-A3B-Instruct (arch Qwen3MoeForCausalLM) from a native modelopt NVFP4 checkpoint via the proven FTW route (ft checkpoint → ft serve, the same path that works for the Qwen3.6-35B-A3B qwen3_5_moe model on this box).
Source tried: NVFP4/Qwen3-Coder-30B-A3B-Instruct-FP4 (modelopt, producer confirmed in hf_quant_config.json).
Repro
ft checkpoint --model <modelopt-NVFP4-dir> --out <out> --dtype bfloat16 --moe-backend offload --gpu 0
Failure
File "freetoken/models/loader.py", line 122, in iter_merged_tensors
File "freetoken/models/qwen3_moe/weight.py", line 57, in sharded_tensors
tensor = shard_tensor(...)
RuntimeError: chunk expects at least a 1-dimensional tensor
(crashes inside the compiled loader.shard_tensor when fed a 0-dim tensor)
Root cause (diagnosed)
qwen3_moe (registered for Qwen3MoeForCausalLM, added in this build) is a generic dense-MoE loader. Its generic experts.<idx>.<name> pattern feeds every per-expert tensor — including the modelopt quant 0-dim scale scalars input_scale and weight_scale_2 — through iter_stacked_experts → shard_tensor, which cannot chunk() a scalar.
By contrast every other modelopt-NVFP4 model here (gemma4, glm5_next, minimax_m2/m3, qwen4_exp) declares an Nvfp4ExpertSourceSpec and routes routed-experts to the offload-bank loader (load_moe_expert_sources), skipping input_scale and carrying weight_scale_2 as the per-row global scale (see glm5_next/weight.py kind_map). qwen3_moe has none of that wiring (only a comment referencing load_moe_expert_sources), so it was evidently validated on bf16 only and cannot ingest a modelopt-NVFP4 checkpoint.
Per-expert tensor shapes from the source:
[] down_proj.input_scale # 0-dim scalar
[2048,384] down_proj.weight
[2048,48] down_proj.weight_scale
[] down_proj.weight_scale_2 # 0-dim scalar
Ask
Wire NVFP4 expert-source support into qwen3_moe — qwen4_exp is the closest template (identical gate_proj/up_proj/down_proj + weight_scale/weight_scale_2/input_scale layout, and it also handles the shared_expert). Or confirm whether Qwen3MoeForCausalLM (Qwen3-Coder-30B-A3B / Qwen3-30B-A3B) NVFP4 is on the roadmap and which source layout it should take.
Happy to help test on sm_120/Windows once a nightly carries it.
Environment
Goal
Serve Qwen3-Coder-30B-A3B-Instruct (arch
Qwen3MoeForCausalLM) from a native modelopt NVFP4 checkpoint via the proven FTW route (ft checkpoint→ft serve, the same path that works for the Qwen3.6-35B-A3Bqwen3_5_moemodel on this box).Source tried:
NVFP4/Qwen3-Coder-30B-A3B-Instruct-FP4(modelopt, producer confirmed inhf_quant_config.json).Repro
Failure
(crashes inside the compiled
loader.shard_tensorwhen fed a 0-dim tensor)Root cause (diagnosed)
qwen3_moe(registered forQwen3MoeForCausalLM, added in this build) is a generic dense-MoE loader. Its genericexperts.<idx>.<name>pattern feeds every per-expert tensor — including the modelopt quant 0-dim scale scalarsinput_scaleandweight_scale_2— throughiter_stacked_experts→shard_tensor, which cannotchunk()a scalar.By contrast every other modelopt-NVFP4 model here (
gemma4,glm5_next,minimax_m2/m3,qwen4_exp) declares anNvfp4ExpertSourceSpecand routes routed-experts to the offload-bank loader (load_moe_expert_sources), skippinginput_scaleand carryingweight_scale_2as the per-row global scale (seeglm5_next/weight.pykind_map).qwen3_moehas none of that wiring (only a comment referencingload_moe_expert_sources), so it was evidently validated on bf16 only and cannot ingest a modelopt-NVFP4 checkpoint.Per-expert tensor shapes from the source:
Ask
Wire NVFP4 expert-source support into
qwen3_moe—qwen4_expis the closest template (identicalgate_proj/up_proj/down_proj+weight_scale/weight_scale_2/input_scalelayout, and it also handles the shared_expert). Or confirm whetherQwen3MoeForCausalLM(Qwen3-Coder-30B-A3B / Qwen3-30B-A3B) NVFP4 is on the roadmap and which source layout it should take.Happy to help test on sm_120/Windows once a nightly carries it.