Repository navigation
added language model embed component - #714
Merged
Merged
Conversation
Collaborator
Author
|
@jakmro added embedding capability for langauge models, tested on lfm, qwen, gemma4. Haven't tested gemma3 |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new decoder_embed_chunk component across the transpile pipeline and runtime, enabling “embedding readout” from causal-LM hidden states (in addition to existing text_embedding encoder-style bundles). It also updates the server to expose embeddings for any bundle that contains the relevant embedding-capable component.
Changes:
- Add
decoder_embed_chunkcomponent specs/adapters for Gemma4, Qwen3, and LFM2-VL to returnlast_hidden_state(nolm_head). - Extend lowering/bundling/planning logic to recognize and emit
decoder_embed_chunk. - Update runtime/server embedding routing to support both
text_embeddingand LM-hidden-state embeddings, and add engine-side pooling/normalization for LM embeddings.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| python/cactus/transpile/model_adapters.py | Adds embed-chunk adapters/specs for multiple model families (returns hidden states instead of logits). |
| python/cactus/transpile/lower.py | Allows internal KV/conv cache lowering for the new decoder_embed_chunk component. |
| python/cactus/transpile/hf_model.py | Ensures decoder_embed_chunk is included in component bundle ordering/manifest writing. |
| python/cactus/transpile/component_plan.py | Includes decoder_embed_chunk in the auto plan for chunked Qwen3 causal-LM pipelines. |
| python/cactus/server.py | Switches embedding capability detection from hardcoded model types to manifest-based component presence. |
| cactus-engine/src/model.cpp | Adds LM-embedding execution path and shared L2-normalization helper. |
| cactus-engine/src/engine.h | Exposes new embedding APIs (get_text_embeddings, get_lm_embeddings) and capability checks. |
| cactus-engine/src/embed.cpp | Routes FFI embedding calls through get_text_embeddings (supports both embedding modes). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| }, | ||
| metadata={"family": family, "task": "causal_lm_logits"}, | ||
| )) | ||
| if "decoder_prefill_chunk" in requested_set: |
Comment on lines
+3308
to
+3312
| reset_component_cache_states(*decoder_embed_); | ||
| decoder_embed_->graph->release_runtime_buffers(); | ||
| decoder_embed_->graph->release_all_weight_pages(); | ||
| unload_component_graph(*decoder_embed_); | ||
| return finalize_pooled_embedding(sum, count, normalize); |
ncylich
added a commit
that referenced
this pull request
Jun 11, 2026
Union resolutions: rope-table externalization (#713) runs before the general materialized-constant externalizer; decoder_embed_chunk (#714) emitted via the unified LFM2 chunked-spec helper and added to the qwen3 text component plan; gemma3 and needle convert families coexist. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
ncylich
added a commit
that referenced
this pull request
Jun 11, 2026
Union resolutions: rope-table externalization (#713) runs before the general materialized-constant externalizer; decoder_embed_chunk (#714) emitted via the unified LFM2 chunked-spec helper and added to the qwen3 text component plan; gemma3 and needle convert families coexist. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.