[Xarbirus]

Overview

Continuation of #30578

Should be merged after #30584 (on the Metal backend the MTP graph yields NaN without a fix)

• implementation of graph_mtp for llama_model_mellum
• converter supports --no-mtp, --mtp and default (combined) GGUFs
• loader supports combined, trunk-only and MTP-only GGUF layouts
• the MTP-layer was trained with SWA
• no regressions observed on Mellum2

Additional information

https://huggingface.co/collections/JetBrains/mellum21 (the model with MTP will be available soon)

Perplexity

llama-perplexity -f wiki.test.raw -c 512 -ngl 99, 591 chunks, all per-chunk values identical:

build	model	PPL
master	public Thinking-BF16	10.3261 +/- 0.07901
branch	public Thinking-BF16	10.3261 +/- 0.07901
branch	combined-BF16	10.3261 +/- 0.07901

Benchmarks

setup: Apple M2 Max, Metal, full GPU offload
llama-server -ngl 99 -c 8192 -np 1 -fa on --jinja [--spec-type draft-mtp --spec-draft-n-max N], 512 generated tokens, temperature 0, mean of 2 runs

Model: combined BF16

Test	Baseline tok/s	MTP 1 tok/s	MTP 1 speedup	MTP 2 tok/s	MTP 2 speedup	MTP 3 tok/s	MTP 3 speedup
code_python	56.31	61.71	1.10x	50.19	0.89x	35.58	0.63x
code_cpp	55.53	60.90	1.10x	46.06	0.83x	36.08	0.65x
code_refactor	55.68	61.03	1.10x	48.41	0.87x	36.16	0.65x
code_review	56.04	58.01	1.04x	44.93	0.80x	37.27	0.67x
sql	56.76	61.37	1.08x	46.37	0.82x	37.51	0.66x
explain_concept	56.62	56.92	1.01x	39.70	0.70x	34.70	0.61x
summarize	56.82	53.59	0.94x	38.98	0.69x	32.94	0.58x
stepwise_math	56.32	59.19	1.05x	46.83	0.83x	41.69	0.74x
translation	56.61	58.75	1.04x	48.29	0.85x	41.13	0.73x
long_code_review	54.01	56.42	1.04x	43.43	0.80x	38.68	0.72x
Mean	56.07	58.79	1.05x	45.32	0.81x	37.17	0.66x

Model: trunk-only BF16 + MTP-only BF16

Test	Baseline tok/s	MTP 1 tok/s	MTP 1 speedup	MTP 2 tok/s	MTP 2 speedup	MTP 3 tok/s	MTP 3 speedup
code_python	58.93	61.25	1.04x	49.17	0.83x	42.13	0.71x
code_cpp	59.68	63.41	1.06x	52.62	0.88x	44.33	0.74x
code_refactor	59.70	63.95	1.07x	55.59	0.93x	44.54	0.75x
code_review	60.11	61.56	1.02x	54.53	0.91x	46.11	0.77x
sql	59.45	64.83	1.09x	56.59	0.95x	45.73	0.77x
explain_concept	59.26	60.05	1.01x	49.54	0.84x	41.54	0.70x
summarize	59.56	59.03	0.99x	47.95	0.81x	39.32	0.66x
stepwise_math	59.43	65.17	1.10x	57.68	0.97x	48.62	0.82x
translation	59.31	64.02	1.08x	61.84	1.04x	50.04	0.84x
long_code_review	57.61	62.14	1.08x	54.71	0.95x	43.10	0.75x
Mean	59.30	62.54	1.05x	54.02	0.91x	44.54	0.75x

Model: trunk-only Q4_K_M + MTP-only Q8_0

Test	Baseline tok/s	MTP 1 tok/s	MTP 1 speedup	MTP 2 tok/s	MTP 2 speedup	MTP 3 tok/s	MTP 3 speedup
code_python	88.36	78.28	0.89x	72.24	0.82x	59.85	0.68x
code_cpp	72.21	73.14	1.01x	69.87	0.97x	56.52	0.78x
code_refactor	74.71	74.01	0.99x	73.09	0.98x	61.10	0.82x
code_review	74.84	73.06	0.98x	64.92	0.87x	55.28	0.74x
sql	73.61	75.88	1.03x	68.40	0.93x	59.67	0.81x
explain_concept	72.72	73.88	1.02x	63.81	0.88x	53.64	0.74x
summarize	72.72	73.26	1.01x	58.63	0.81x	49.45	0.68x
stepwise_math	73.57	76.47	1.04x	72.41	0.98x	63.95	0.87x
translation	73.47	77.27	1.05x	67.98	0.93x	55.34	0.75x
long_code_review	71.85	72.21	1.01x	67.09	0.93x	56.83	0.79x
Mean	74.80	74.75	1.00x	67.84	0.91x	57.16	0.76x

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, understanding the graph, refactoring, benchmark results