Skip to content

qwen35 moe#258

Open
wine99 wants to merge 1 commit into
dev_backend_openvinofrom
qwen35-moe
Open

qwen35 moe#258
wine99 wants to merge 1 commit into
dev_backend_openvinofrom
qwen35-moe

Conversation

@wine99

@wine99 wine99 commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

@cavusmustafa Changes from your PR #221 included here

  1. when use_bias=true, use (w - zp_f16) * scale instead of w * scale + bias_f16). For quantized expert weights, use_bias is always true
  2. changes in other ops (eg process_view)

Differences from your PR:

  1. use rank-3 shape for expert weights after dequantaztion subgraph (ie rank-4 before dequant) instead of rank-2. So that both CPU and GPU recognize the dequantization pattern by default and can fuse to GatherMatmulGather
  2. copied GatherMatmul header file and construct it directly, like what did with GatedDeltaNet
  3. for CPU, this PR [CPU][Snippets] Skip tokenizing GatherMatmul's dequantization weights path openvinotoolkit/openvino#36937 is needed
  4. for CPU and GPU [CPU][GPU] Recognize GatherMatmul as a decompression-multiply consumer openvinotoolkit/openvino#36936 is needed

Both PRs in ov are needed. Without the first one, the dequantization subgraph will be converted to a snippet in ov cpu, preventing GatherMatmulCompressed fusion. Without the second one, ov::pass::KeepConstPrecision will not be called and u4->u8 conversion will happen on both CPU and GPU. Once the the second PR is merged, we can also remove the copied mark_dequantzation_subgraph.hpp

@wine99
wine99 marked this pull request as ready for review July 20, 2026 09:00
@wine99
wine99 requested a review from cavusmustafa as a code owner July 20, 2026 09:00
@wine99

wine99 commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator Author

current status:

  • test_backend_ops -o "MUL_MAT_ID" pass
  • GatherMatmulCompressed op seen in CPU and GPU exec graph with a small synthetic qwen35 moe model
  • llama-simple Qwen3.6-35B-A3B-UD-Q4_K_S OOM i7-14700K 64GB ubuntu throughput is 4.98 tokens/second.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant