fix(template): preserve native user follow-ups after tool results

[Excelius-Wang]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

A valid assistant → tool → user → assistant sequence raises AssertionError: response_role: "user" in the Swift pairwise encoder. The user sample in #8499 reproduces this when a follow-up arrives after tool results. This revision moves detection from the Qwen template into common Template preprocessing and delegates native turn boundaries to small agent-specific formatters, reusing existing tool-result rendering, the pairwise encoder and assistant loss handling.

Supported text paths include Hermes/Qwen, GLM4.5/4.7/5.1, DeepSeek V3.1, Youtu and Hunyuan. Consecutive tool results and user follow-ups retain their boundaries; literal template placeholders in user/tool text are not interpreted again. ReAct handling is left on its existing path.

Multimodal support is limited to processor-validated Qwen3.5 images/frame-list video (including multiple images, multiple videos and mixed image/video turns), and Qwen2.5-Omni with the Hermes agent for audio-only media inputs. Other media combinations remain excluded.

Two related Hunyuan inconsistencies also needed correction: already-serialized calls could lack model template metadata, and structured calls inherited Hermes JSON serialization although the native parser expects a function name and fenced JSON. These now use consistent native formatting. When chat templates are disabled, Hermes/Youtu honor the existing absent-separator convention instead of calling .copy() on None; this preserves plain generation behavior. This does not add non-chat tool/user-follow-up support or change general consecutive-user handling without preceding tool results.

Refs #8499, #7896. Related maintainer draft: #10201; this revision follows its common-base direction and keeps the overlap explicit.

Experiment results

• 24 regression test methods passed with no skips both locally and with the complete patch applied to upstream main 494d9cd9043113aa356554ae775946e718b9c8ab. Coverage includes 84 media subcases, 96 offline/real-tokenizer Hunyuan sequence cases, and 12 non-chat combinations that failed before the separator correction and pass afterward. Tests check complete token IDs, assistant labels, media features/grids, tool-call representations, response weights, literal placeholders and input immutability.
• The original issue sample encodes successfully, retaining the follow-up without supervising it. This uses a Qwen3.5-2B processor with the same template family, not the reporter's 27B model weights.
• DeepSeek: 16 comparisons against the pinned official V3.1 template check complete rendered text and token IDs; additional comparisons check native tool/user boundary fragments in vLLM's reference template (not full vLLM text equality).
• Hunyuan: real tencent/Hunyuan-1.8B-Instruct tokenizer at de940610ab5d45b4fedbd37e08444d0dc502837c. Four ordinary cases match current official text/IDs. Its current Jinja leaves the tool-result block unclosed before a user follow-up, so four follow-up comparisons use explicitly canonicalized historical official role rendering plus native tool wrapping. These are not claimed to match the malformed current official follow-up output.
• Full pre-commit run --all-files and git diff --check passed.

Reproduce the regression suite from the repository root with local processor paths:

export SWIFT_TEST_QWEN_PROCESSOR=/path/to/Qwen3.5-processor
export SWIFT_TEST_OMNI_PROCESSOR=/path/to/Qwen2.5-Omni-processor
export SWIFT_TEST_HUNYUAN_PROCESSOR=/path/to/Hunyuan-1.8B-tokenizer
python -m unittest discover -s tests/utils -p 'test_tool_user*.py' -v
pre-commit run --all-files

Without the corresponding asset paths, real-processor integration tests skip; offline tests still run. Validation here is CPU encoding/processor validation, not full model-weight training. No GPU training, packing/sequence-parallel validation, arbitrary multimodal combinations or encoded video-file decoding support is claimed.

[tastelikefeet]

Two comments:

1. Why ignore inputs.is_multimodal?
2. Not only qwen has this problem, I think move this to the base class is correct:
   https://github.com/modelscope/ms-swift/pull/10201/changes
   This PR comes from your code, may be can help you.

[Excelius-Wang]

Thanks for the reference to #10201. I updated this PR to address both points:

1. Tool-result/user-follow-up detection now lives in the common Template preprocessing loop. Agent-specific hooks preserve native formatting while reusing the existing tool-result formatters and pairwise encoder. The text coverage includes Hermes/Qwen, GLM, DeepSeek V3.1, Youtu and Hunyuan.
2. Multimodal inputs are no longer excluded wholesale. Processor tests cover multiple images, two audio clips, two frame-list videos and mixed image/video turns, comparing token IDs, media features and assistant labels. The gate is limited to the verified Qwen3.5 image/video and Qwen2.5-Omni Hermes audio paths.

The 24-method regression suite passes without skips locally and on upstream main 494d9cd plus the patch; full pre-commit also passes. Tests also cover the non-chat separator regression caught during review. The PR description records asset-dependent CI skips and the Hunyuan oracle limitation: its current official follow-up Jinja is malformed, so corrected follow-up boundaries use a qualified historical-template comparison rather than claiming exact current-template agreement.

I saw that #10201 is postponed pending the refactor. This update keeps the implementation and evidence available for that work; happy to adapt it to the refactored interface.