Skip to content

perf(KV-FP8): serve fp8 KV prefill through the bf16 dense dequant - #3360

Open
iiLaurens wants to merge 3 commits into
mudler:mainfrom
iiLaurens:row/KV-FP8
Open

perf(KV-FP8): serve fp8 KV prefill through the bf16 dense dequant#3360
iiLaurens wants to merge 3 commits into
mudler:mainfrom
iiLaurens:row/KV-FP8

fix(KV-FP8): release the fp8 dense scratch on every unwind path

b2f195e
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs