Skip to content

perf(gemma4): fuse routed expert GELU gating - #764

Open
pculaf wants to merge 1 commit into
Luce-Org:mainfrom
pculaf:perf/gemma4-routed-geglu
Open

pculaf wants to merge 1 commit into
Luce-Org:mainfrom
pculaf:perf/gemma4-routed-geglu

Conversation

@pculaf

@pculaf pculaf commented Sep 24, 2026 •

Copy link
Copy Markdown

Replace two tensor copies, GELU, and multiplication in Gemma 4's routed experts
with one ggml_geglu operation. This computes the same GELU(gate) * up result
using an existing kernel, reducing four GPU kernel executions to one per MoE
layer. For Gemma 4 26B-A4B, that removes 90 kernel executions per ordinary decode
step.

The change covers only routed experts. A shared-expert variant showed no
measurable benefit in preliminary testing, so it is excluded.

Tested on NVIDIA A10 with Gemma 4 26B-A4B IT Q4_K_M, a Q8_0 Gemma DFlash
draft, F16 KV and temperature 0. Prompt caching was disabled, and performance
timings were collected without a GPU profiler running.
Benchmark revision:
97d7c74.

The server's context capacity was 2048 tokens per request, covering input plus
generated output. Actual requests were shorter:

  • Coding: all ten prompts from
    bench_he.py,
    sent through Gemma's chat template: 100–161 input tokens and 256 output tokens.
  • Math: the first five cases from
    bench_gsm.jsonl,
    with 44–126 input tokens and a 1024-token output cap. Actual outputs were
    150–1024 tokens in ordinary mode and 195–1024 with DFlash.

Input counts include the chat template. Generation could end before its output cap;
the largest input-plus-output length observed was 1150 tokens.

Each mode used A/B/B/A order on one GPU (A: original, B: patched). Each session
included one full-corpus warmup pass followed by two measured passes, giving
four measurements per variant per prompt. Times below sum the per-prompt
median decode durations; speedup is original / patched - 1.

Mode Prompts Original (ms) Patched (ms) Speedup
Ordinary Coding 29426.85 29362.55 +0.22%
Ordinary Math 49301.40 48892.75 +0.84%
DFlash Coding 15223.15 14928.10 +1.98%
DFlash Math 16313.55 16114.85 +1.23%

DFlash gains were consistent across both A/B pairs. Ordinary results were
inconclusive: four coding prompts were slightly slower, and the math gain
varied from 1.40% to 0.09% between pairs.

All 240 timed responses and 120 warmups preserved the original outputs, token
counts and speculative acceptance within each mode.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant