TokenFurnace

Performance tuning and benchmarks for Hugging Face models on four NVIDIA Quadro RTX 8000 GPUs.

The workstation has:

• 4x NVIDIA Quadro RTX 8000, 48 GiB each
• NVLink pairs between GPUs 0-1 and 2-3
• 192 GiB aggregate VRAM
• Intel Xeon W-2295 and 256 GiB system RAM

The repository is organized by hardware, model target, runtime backend, and tuning profile rather than around one model. Model weights are downloaded separately and are not committed to Git.

What's in this repository

• Hardware documentation records the GPU topology, NVLink pairings, firmware, drivers, CUDA capabilities, and installed inference runtimes.
• Model targets define Hugging Face sources, quantizations, GPU placement, runtime-specific settings, and recommended serving profiles.
• Benchmark reports preserve measured native llama.cpp and OpenAI-compatible API results, including unsuccessful configurations where they help explain the selected defaults.
• Automation scripts download model artifacts, build runtime variants, launch servers, run repeatable benchmarks, inspect status, and connect local models to coding agents.

Documentation

Start with the four-RTX-8000 hardware profile for the constraints shared by every experiment. Each model then has a target guide describing its artifacts and profiles, plus a separate performance report containing measurements and conclusions.

Model	Target guide	Performance report	Backend and status
DeepSeek V4 Flash 0731	Setup and profiles	Q4 and DSpark results	llama.cpp + DSpark; optimized and measured
Qwen3-Coder-Next 80B-A3B	Setup and profiles	Measured and projected results	llama.cpp Q4_K_M measured; vLLM FP16 projected
Qwen3.6-35B-A3B	Setup and profiles	Q4_K_M and MTP results	llama.cpp measured; vLLM pending
Qwen3.8-27B	Setup and profiles	Topology and runtime results	llama.cpp Q4_K_M and official FP16 vLLM; optimized and measured
Qwen3.8-27B Uncensored OrcaRouter	Setup and profiles	Tensor split and MTP results	llama.cpp Q4_K_M + MTP-2; optimized and measured
Qwen3.8-Flash-Next	Setup and profiles	Two- and four-GPU results	llama.cpp Unsloth Q3_K_XL; optimized and measured
GLM-5.3-Flash	Setup and profiles	Four-GPU results	llama.cpp Unsloth UD-Q3_K_XL; measured
Laguna S 2.1	Setup and profiles	Baseline and DFlash results	llama.cpp Q4_K_M; optimized and measured
Muse Glimmer 30B	Setup and profiles	Baseline and DFlash results	llama.cpp dynamic Q4_K_XL; optimized and measured
Spark-X2.5-4B	Setup and profiles	Single-GPU Q8 results	XHToken llama.cpp Q8_0; measured
Ternary Bonsai 27B	Setup and profiles	Single-GPU ternary results	PrismML llama.cpp PQ2_0; measured
Gemma 4 12B	Setup and profiles	Text and vision results	llama.cpp Q4_K_M plus F16 projector; measured
Nex-N2.5-mini	Setup and profiles	Text and vision results	llama.cpp Q4_K_M plus F16 projector; measured
MiniMax-M2.5	Setup and profiles	Two-GPU Q2 results	llama.cpp Q2_K on one NVLink pair; measured

Repository layout

hardware/<profile>/         Workstation topology, firmware, and runtime support
targets/<model>/README.md   Model source, artifacts, profiles, and usage
targets/<model>/PERFORMANCE.md
                            Benchmark conditions, results, and conclusions
targets/<model>/profiles/   Reproducible runtime configuration
scripts/backends/           Runtime-specific launch implementations
scripts/serve-model.sh      Generic target/profile launcher
scripts/download-model.sh   Generic Hugging Face downloader
scripts/benchmark-model.sh  OpenAI-compatible API benchmark
scripts/benchmark-native.sh llama.cpp prefill/decode kernel benchmark

Claude slash commands

• /document-hardware — invoke when auditing or refreshing the workstation's hardware, firmware, topology, CUDA, runtime, and compatibility documentation.
• /benchmark-model — invoke when adding or benchmarking a model target, comparing profiles or GPU topologies, or documenting measured performance.

Usage

DeepSeek balanced profile:

./scripts/serve-model.sh deepseek-v4-flash-0731 q4-balanced

Qwen fastest single-request decode (llama.cpp Q4_K_M, 2-GPU NVLink pair — the default):

./scripts/download-model.sh qwen3-coder-next-80b-a3b llama-cpp-q4km-2gpu
./scripts/serve-model.sh qwen3-coder-next-80b-a3b            # default = llama-cpp-q4km-2gpu (GPUs 0,1)

Qwen explicit 4-GPU long-prefill throughput profile (llama.cpp Q4_K_M):

./scripts/serve-model.sh qwen3-coder-next-80b-a3b llama-cpp-q4km

Qwen highest-throughput profile (vLLM FP16, NVLink-pair-aware TP2 x PP2):

./scripts/download-model.sh qwen3-coder-next-80b-a3b vllm-fp16-tp2pp2
./scripts/serve-model.sh qwen3-coder-next-80b-a3b vllm-fp16-tp2pp2

Qwen3.6 fastest interactive profile (single GPU, Q4_K_M, MTP):

./scripts/download-model.sh qwen3.6-35b-a3b
./scripts/serve-model.sh qwen3.6-35b-a3b

Qwen3.8 fastest profile (2-GPU NVLink tensor split, Unsloth Q4_K_M):

./scripts/download-model.sh qwen3.8-27b
./scripts/benchmark-native.sh qwen3.8-27b
./scripts/serve-model.sh qwen3.8-27b

Qwen3.8 official unquantized checkpoint (official BF16 safetensors cast to native Turing FP16 on load):

./scripts/download-model.sh qwen3.8-27b vllm-official-fp16-tp2
./scripts/serve-model.sh qwen3.8-27b vllm-official-fp16-tp2-mtp1

GLM-5.3-Flash UD-Q3_K_XL on all four GPUs:

./scripts/download-model.sh glm-5.3-flash
./scripts/serve-model.sh glm-5.3-flash

Laguna S 2.1 Q4_K_M on all four GPUs:

./scripts/download-model.sh laguna-s-2.1
./scripts/serve-model.sh laguna-s-2.1

Muse Glimmer 30B dynamic Q4_K_XL with DFlash on one GPU:

./scripts/download-model.sh muse-glimmer-30b
./scripts/serve-model.sh muse-glimmer-30b

Benchmark a running target (single-request latency, or aggregate throughput):

./scripts/benchmark-model.sh qwen3-coder-next-80b-a3b
./scripts/benchmark-model.sh qwen3-coder-next-80b-a3b llama-cpp-q4km -- --prompt-file PROMPT.txt --max-tokens 1
./scripts/benchmark-model.sh qwen3-coder-next-80b-a3b vllm-fp16-tp2pp2 -- --concurrency 16 --requests 64

Run the standard native llama.cpp sweep without starting a server:

./scripts/benchmark-native.sh qwen3.6-35b-a3b llama-cpp-q4km-1gpu

The prompt-rate fields are end-to-end API rates. With --max-tokens 1 and a long prompt they are useful prefill-oriented comparisons, but they are not server-internal kernel timings.

Point coding agents at the local server (both wrappers default to the Qwen target and its 2-GPU default profile on port 8090):

./scripts/copilot-local.sh   # OpenAI /v1, Qwen default (either backend)
./scripts/claude-local.sh    # Anthropic API, Qwen default (llama.cpp)

Inspect status:

./scripts/status.sh deepseek-v4-flash-0731
./scripts/status.sh qwen3-coder-next-80b-a3b

Downloaded model weights remain outside Git under /home/algore/models.