Skip to content

docs: add validated Jetson Thor DFlash2 serving setup - #164

Open
marcodelpin wants to merge 1 commit into
z-lab:mainfrom
marcodelpin:docs/jetson-thor-dflash2-256k
Open

marcodelpin wants to merge 1 commit into
z-lab:mainfrom
marcodelpin:docs/jetson-thor-dflash2-256k

Conversation

@marcodelpin

Copy link
Copy Markdown

Summary

  • document a tested Qwen3.8-27B DFlash 2 configuration for the 128 GB Jetson AGX Thor Developer Kit
  • pin the validated vLLM, NVIDIA PyTorch, FlashInfer, CUDA architecture, weight, and KV-cache formats
  • show explicit KV allocation for a 262,144-token context limit
  • clarify how --kv-cache-memory-bytes and --gpu-memory-utilization interact in the tested vLLM revision

Validation

The upstream package workflow was reproduced locally on Python 3.12.13:

  • built the sdist and wheel
  • passed twine check --strict
  • installed the generated wheel
  • imported dflash
  • smoke-tested dflash, dflash generate, and dflash benchmark

The documented configuration was also checked on a live 128 GB Jetson AGX Thor:

  • vLLM V2 model runner active
  • DFlash2Qwen3ForCausalLM draft model loaded
  • text generation and health check successful
  • FP8 E4M3 KV cache on sm_110
  • max_model_len=262144
  • 453,669 tokens of KV capacity, reported as 1.73 concurrent sequences at the 262,144-token limit

The 256K statement describes allocated KV capacity; it does not claim that a 256K-token prompt was executed.

AI assistance disclosure

This documentation change was AI-assisted and has not yet received independent human code review. The commands, package checks, runtime configuration, and reported capacity were verified against the tested hardware and final diff.

@marcodelpin
marcodelpin marked this pull request as ready for review August 27, 2026 19:51
@marcodelpin

Copy link
Copy Markdown
Author

Hi - the "Package checks" workflow on this PR is sitting at action_required, so CI hasn't run yet. Could a maintainer approve the run when you get a chance?

It's a docs-only change (README.md), so the wheel build and CLI smoke tests should be unaffected - happy to rebase on current main first if you'd prefer.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant