fix(lisa): select language layers in multimodal models

[taking-lying-flat]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

LISA currently selects the first nn.ModuleList in the entire model. In Qwen2.5-VL, this is model.visual.blocks, so LISA switches vision blocks while leaving every language layer trainable. If vision parameters were frozen before optimizer creation, the callback can also enable gradients on vision parameters that are absent from the optimizer.

Resolve candidate layers using the existing model architecture metadata: prefer module_list when specified, otherwise restrict the search to language_model prefixes. Exclude registered vision, alignment, and generation components, including those nested under a language-model prefix. Unwrap parallel/compiled models before resolving these paths.

The change is limited to swift/callbacks/lisa.py (+13/-1). Layer switching still begins after optimizer creation. This addresses layer-container selection; it does not change freezing policies inside the selected language layers.

Scope: the fix uses registered architecture information. Models without that information retain the existing fallback; ambiguous multi-backbone architectures need separate validation.

Experiment results

Used an actual Transformers Qwen2_5_VLForConditionalGeneration with a tiny random configuration (4 vision blocks, 8 language layers), lisa_activated_layers=2, and an optimizer constructed before on_train_begin:

Initial vision state	Version	Trainable language layers	Trainable vision blocks	Trainable parameter tensors absent from optimizer
Trainable	Before	8	2	0
Trainable	After	2	4	0
Frozen	Before	8	2	24
Frozen	After	2	0	0

The missing optimizer entries indicate inconsistent gradient/optimizer state, not that those vision parameters were necessarily updated.

20 tests passed locally:

• 9 Qwen2.5-VL cases cover trainable/frozen/partially frozen vision parameters and unwrapped/DataParallel/compiled models. Check selected layers, repeated switches, non-target gradient flags, and optimizer membership.
• 5 cases cover different registered language-module layouts and explicit layer paths.
• 2 CPU HF Trainer runs with a tiny multimodal test model verify actual optimizer updates across three different layer selections, with vision either frozen or trainable.
• All 4 existing LISA tests pass. The frozen-vision Qwen2.5-VL regression fails against the original callback.

Also checked the actual registered Transformers model structures on the meta device for Qwen2.5-VL, Qwen3-VL, Qwen3.5, Gemma3-Vision, Gemma4, Gemma4 Unified, Llama3.2-Vision, and Llama4. All eight resolve to their language decoder layer stack; these are structural checks, not training tests for those models.

Changed-file pre-commit and git diff --check pass. Targeted regression scripts were run locally; this PR contains only the production fix. Validation used Transformers 5.14.1 and did not include pretrained-model or distributed training runs. The checks do not establish support for every registered model or Transformers version.