Repository navigation
Fix multi-turn chat and tool-call defects across model families - #721
Merged
Merged
Conversation
Any second chunked prefill on a gated-deltanet bundle threw a state rank error because moving the cache buffer into the decoder graph left the prefill node's BufferDesc empty. Restore the source to a fresh unallocated descriptor with its original shape and precision so the next execute re-initializes the state exactly as on first run. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Every generation after a force_tools turn on the same handle degenerated into <|tool_call> loops on gemma models, surviving cactus_reset. The deactivating init left the previously computed force bias in place while update() no-ops when inactive, so decode kept force-picking the call-start token. Clear the bias in init and drop the redundant reset in clear_tool_constraints that recomputed it just before deactivation. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Adds a cactus_render_prompt FFI call that exposes the fully rendered chat prompt, plus a ctypes wrapper and export. A golden pytest suite checks engine renders and tokenizations against the HF reference chat templates for gemma-4, qwen3, and lfm2 across system, tool-call, parallel-tool-call, and thinking conversations, plus the gemma-4 image expansion. Aligns the gemma-4 formatter with the reference template: <|think|> gains its trailing newline and the image block drops its surrounding blank lines. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
The internal KV-cache layer key fell through to the node id whenever attention_layer_index was 0, and decoder_media_step graphs never received attention_layer_index hints at all, so components of one bundle disagreed on layer keys and the engine's positional cache-state pairing failed at load time with "prefill and step cache layer mismatch". Derive the key with explicit None checks before falling back to the node id, and include decoder_media_step in the attention-hint assignment. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
A stale or revoked API key made every low-confidence turn re-attempt the cloud handoff, burning up to cloud_timeout_ms per turn before falling back to local output. Track http_401/http_403 completion failures on the model handle and drop handoff eligibility for the rest of the session once one is seen; timeouts, 5xx, and connection errors keep retrying as before. The pre-streaming trigger now also respects handoff eligibility, and the live test drives a forced handoff against a local 401 server to verify the second turn stays local. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
The engine's parse_tools_json scans for a literal "function" key, so
the flat {name, description, parameters} dicts _translate_tools emitted
were parsed as zero tools and requests silently lost their tool
definitions. Build the wrapped form once and filter it for the
tool_choice object branch.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Gemma3-family prompts now follow the FunctionGemma chat template:
system text and tool declarations render in a developer turn, prior
assistant tool calls as <start_function_call>call:name{...} blocks,
and tool results as <start_function_response> blocks, with <escape>
quoting instead of the gemma4 pipe tags. The tool-call constrainer
keys its tags on the model family and forces multi-token tags one
token at a time, so force_tools no longer degenerates into marker
loops on bundles whose tags are not single special tokens. DSL-to-
JSON argument conversion escapes string content so malformed model
output cannot corrupt the response payload.
Signed-off-by: Noah Cylich <noahcylich@gmail.com>
format_qwen_style appended the Qwen3-hybrid "<think>\n\n</think>\n\n" generation scaffold (and think-wrapped trailing assistant prefills) for every qwen-family bundle. Non-thinking bundles such as qwen3-vl-2b-instruct have no think handling in their chat template, so the scaffold pushed the model off-distribution: at temperature 0 the second turn of a conversation greedily replayed the previous assistant message byte-for-byte instead of answering the new question. The dequantized bundle reproduces both behaviors in the reference stack: with the scaffold it emits the identical replay, without it it answers the question. Gate the think emission sites on the bundle's chat template actually containing <think>, which keeps hybrid-thinking renders byte-identical, and add a multi-turn distinctness regression test. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
ncylich
force-pushed
the
fix/e2e-fleet-fixes
branch
2 times, most recently
from
June 13, 2026 00:18
9097572 to
819a0ce
Compare
Moves bundle discovery into tests/bundles.py, folds the repeated chunked-prefill check into the llm suite reusing its existing helper, drops import-time HF env mutation for local_files_only, and extracts the shared tool-capture setup in the server tests. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
ncylich
force-pushed
the
fix/e2e-fleet-fixes
branch
from
June 13, 2026 00:20
819a0ce to
f2631e6
Compare
… test The rejection test ran against the default LLM fixture (gemma-4-e2b-it), which carries a decoder_embed_chunk component and therefore legitimately serves hidden-state embeddings, so the endpoint returned 200 instead of the expected 400. Discover a bundle that lacks both text_embedding and decoder_embed_chunk, skipping when none is present, and share the server startup via a context manager. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
The gemma4 FP16 weight scaler keyed the 1/16 down-scaling on the filename embed_vision_embedding.weights, but the vision embedding projection is saved as embed_vision_proj.weights, so it was stored unscaled. The gemma4 runtime operates in a 1/16-scaled residual space (token embeddings, norms, etc. are all down-scaled to keep activations in fp16 range), so an unscaled projection left image soft-tokens ~16x too large in the residual stream. This drowned out the per-layer refinement of image features and degraded multimodal reasoning: gemma-4-e4b-it confidently misidentified a monkey as a toucan, while e2b was unaffected only because its projection was converted before the filename was dropped from the scale set (regression in #659). Add the real filename so the projection is down-scaled like every other gemma4 embedding-space weight. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Same bug class as the vision embedding projection: the gemma4 audio feature projection (model.embed_audio.embedding_projection.weight -> embed_audio_proj.weights) maps audio soft-tokens into the LM's 1/16-scaled residual space but was stored unscaled (scale_factor 1.0, confirmed: stored norm == HF norm), leaving audio soft-tokens ~16x too large in the residual stream. Add embed_audio_proj and embed_audio_embedding to the gemma4 down-scale set so audio embeddings are scaled like vision embeddings and every other embedding-space weight. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
…ion" This reverts commit f892740.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes every engine/transpiler/server defect found by a full multi-turn chat + tool-call E2E sweep across all supported model families. One commit per fix; every fix landed with a failing regression test first. Final fleet validation on this branch: 11/11 bundles meet baseline — no crashes, no degeneration loops, no marker leakage.
Commits
GATED_DELTANET expects state rank 4); the moved-from cache node was left with an empty descriptor. New graphcachetest + enginerecurrent_prefillsuite.force_toolsturn, every later unconstrained generation on the handle force-picked<|tool_call>(survivedcactus_reset), degenerating gemma-4 tool-result and image turns into token loops. Root cause was a stale constraint logit bias, not the chat template.cactus_render_promptFFI + pytest diffing engine renders and token ids againstapply_chat_templatefor gemma-4/qwen3/lfm2/functiongemma (chat, tools, tool responses, parallel calls, thinking, images). Caught two real gemma-4 render divergences, fixed here. Template drift can no longer land silently.or-chain to node-id keys anddecoder_media_stepwas excluded from attention-hint assignment, crashing qwen3-vl image turns withprefill and step cache layer mismatch.http_401/403now disables handoff for the session. Live mock-server pytest.<start_function_call>-token contract. FunctionGemma now passes all tool scenarios; base gemma-3 no longer garbles.<think>scaffold was appended to every QWEN bundle; qwen3-vl (no thinking template) went off-distribution and parroted its previous turn byte-for-byte at temp 0. Proven bidirectionally against the dequantized checkpoint in PyTorch._translate_toolssent flat dicts the engine silently parsed as zero tools;cactus servetool calling never worked. Live A/B:finish_reasonstop→tool_calls.Validation
llm,cache,recurrent_prefill, prefix-cache tests green across qwen3-0.6b/1.7b, qwen3.5, gemma-4, default LFM2 (one pre-existingtool_multiple_tool_call_invocationsflake, identical on main).