Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace two tensor copies, GELU, and multiplication in Gemma 4's routed experts
with one
ggml_gegluoperation. This computes the sameGELU(gate) * upresultusing an existing kernel, reducing four GPU kernel executions to one per MoE
layer. For Gemma 4 26B-A4B, that removes 90 kernel executions per ordinary decode
step.
The change covers only routed experts. A shared-expert variant showed no
measurable benefit in preliminary testing, so it is excluded.
Tested on NVIDIA A10 with Gemma 4 26B-A4B IT Q4_K_M, a Q8_0 Gemma DFlash
draft, F16 KV and temperature 0. Prompt caching was disabled, and performance
timings were collected without a GPU profiler running.
Benchmark revision:
97d7c74.The server's context capacity was 2048 tokens per request, covering input plus
generated output. Actual requests were shorter:
bench_he.py,sent through Gemma's chat template: 100–161 input tokens and 256 output tokens.
bench_gsm.jsonl,with 44–126 input tokens and a 1024-token output cap. Actual outputs were
150–1024 tokens in ordinary mode and 195–1024 with DFlash.
Input counts include the chat template. Generation could end before its output cap;
the largest input-plus-output length observed was 1150 tokens.
Each mode used A/B/B/A order on one GPU (A: original, B: patched). Each session
included one full-corpus warmup pass followed by two measured passes, giving
four measurements per variant per prompt. Times below sum the per-prompt
median decode durations; speedup is
original / patched - 1.DFlash gains were consistent across both A/B pairs. Ordinary results were
inconclusive: four coding prompts were slightly slower, and the math gain
varied from 1.40% to 0.09% between pairs.
All 240 timed responses and 120 warmups preserved the original outputs, token
counts and speculative acceptance within each mode.