Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions examples/hf_ptq/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,8 @@ scripts/huggingface_example.sh --model $HF_PATH --quant <QFORMAT> --tp [1|2|4|8]

> *FP8 calibration over a large model with limited GPU memory is not recommended but possible with the [accelerate](https://huggingface.co/docs/accelerate/en/usage_guides/big_modeling) package. Please tune the device_map setting in [`example_utils.py`](./example_utils.py) if needed for model loading and the calibration process can be slow.*

> *Models with linear-attention layers (Qwen3-Next, Qwen3.5, Qwen3.8, GLM-5.x Flash, Kimi Linear, OLMo Hybrid) run those layers through Transformers' slow PyTorch reference kernels unless the [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) kernels are installed: `pip install "fla-core>=0.5.2"` (earlier versions compute packed Gated DeltaNet sequences wrongly). Install it only where these models run on GPU, since Transformers then uses the kernels for CPU tensors too, and they fail there. On GLM-5.3-Flash over 4x GB300 they more than halved layerwise GPTQ calibration, mostly because the fused kernels need far less memory per sample, so a larger calibration batch fits.*

> *Huggingface models trained with `modelopt.torch.speculative` can be used as regular Huggingface models in PTQ. Note: there is a known issue with Huggingface models loaded across multiple GPUs for inference (i.e., "Expected all tensors to be on the same device, but found at least two devices..."). When encountered this error in PTQ of speculative decoding models, try reducing the number of GPUs used.*

> *Calibration by default uses left padding_side for the Huggingface tokenizer as it usually leads to lower accuracy loss. The exported tokenizer files restores the default padding_side.*
Expand Down
1 change: 1 addition & 0 deletions plugins/modelopt/skills/ptq/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,7 @@ Report the gate result before moving on. Follow the canonical report format and
## Common Pitfalls

- **Model-specific dependencies**: Models with `trust_remote_code` may import packages not in the container (e.g., `mamba-ssm` for hybrid Mamba models). See Step 2.5. Use `EXTRA_PIP_DEPS` env var with the launcher, or install manually before running `hf_ptq.py`
- **Linear-attention models** (Qwen3-Next/3.5/3.8, GLM-5.x Flash, Kimi Linear, OLMo Hybrid) fall back to slow, memory-hungry PyTorch kernels without `pip install "fla-core>=0.5.2"`; install it before calibrating on GPU (it breaks CPU runs of these models)
- **Transformers version**: New models may need a newer version of transformers than what's installed. Check `config.json` for `transformers_version`. In containers, beware of `PIP_CONSTRAINT` blocking upgrades — see `references/slurm-setup-ptq.md` for workarounds

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bot comment.

[P2] Quote the versioned requirement: pip install "fla-core>=0.5.2". In Bash/POSIX shells, the current > redirects stdout to a file named =0.5.2, and pip receives only fla-core. An already-installed older version therefore satisfies the command and is not upgraded, defeating the stated minimum. Markdown backticks do not provide shell quoting.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 58d6763: the skill now says pip install "fla-core>=0.5.2", quoted, and the hf_ptq README note gets the same quoted pin plus the reason for it (earlier versions compute packed Gated DeltaNet sequences wrongly).

- **Gated datasets**: Some calibration datasets require HF authentication. Set `HF_TOKEN` in the job environment. Use `--dataset cnn_dailymail` only as the constrained-environment fallback described in Step 4, not as the preferred calibration set
- **NFS root_squash + Docker**: See the common skill's `slurm-setup.md` section 5
Expand Down
Loading