QwenImage21Pipeline: image editing ignores the instruction and returns a haloed near-copy at output_resolution=1024 (default); 512/768 edit correctly

[xocialize]

Describe the bug

With QwenImage21Pipeline, image editing at the default output_resolution=1024 ignores the instruction and returns an over-sharpened, haloed, over-saturated near-copy of the input image. The same call with output_resolution=512 or 768 produces a correct, clean edit.

This does not depend on the prefix KV cache (use_kv_cache=False gives the same image), on guidance (true_cfg_scale=4.0 adds a red band where the scarf should be but keeps the halo), on the transformers version (5.14.1 and 5.17.0 produce bit-identical results), or on precision (fp32 end to end gives the same image as bf16 to 43 dB PSNR).

An independent re-implementation of this pipeline in another framework (Swift/MLX, block-level parity with the diffusers transformer at fp32) reproduces the 1024² output from the same noise to 36 dB (bf16) / 54 dB (fp32) PSNR, so the behaviour follows from the pipeline's math as written rather than from a backend. I have only tested on Apple silicon (MPS), so a CUDA confirmation would be useful.

Observed at output_resolution=1024 for two prompts ("Make the dog wear a red scarf", "Change the background to a sunset beach") on a natural photo generated by the pipeline itself, and for a synthetic image; at 320² and 512²/768² the same edits are correct.

Questions this raises, in case they help locate it:

1. Is 1024² (4,096 condition + 4,096 target tokens) inside the trained editing regime, or should output_resolution default lower / be capped for edits?
2. QwenImage21Rope advances the frame position by max(h, w) after each image block, so the condition→target frame offset grows with the grid (20 at 320², 64 at 1024²). Is that the training convention?
3. calculate_shift is fed latents.shape[1] (target tokens only) while the joint sequence is twice that for a 1-image edit. Intended?

Reproduction

import torch
from diffusers import QwenImage21Pipeline

pipe = QwenImage21Pipeline.from_pretrained("Qwen/Qwen-Image-2.1", torch_dtype=torch.bfloat16).to("mps")  # or "cuda"

# 1) a natural photo from the model itself
photo = pipe(
    prompt="A photo of a golden retriever sitting on a wooden dock by a calm lake, soft morning daylight, shallow depth of field",
    width=1024, height=1024, num_inference_steps=40,
    generator=torch.Generator("cpu").manual_seed(42),
).images[0]
photo.save("photo.png")

# 2) the same edit at three output resolutions
for res in (512, 768, 1024):
    img = pipe(
        prompt="Make the dog wear a red scarf",
        image=photo, output_resolution=res, num_inference_steps=40,
        generator=torch.Generator("cpu").manual_seed(42),
    ).images[0]
    img.save(f"scarf_{res}.png")

# variants that do NOT change the 1024 result:
#   use_kv_cache=False
#   true_cfg_scale=4.0, negative_prompt=" "   (partial: a red band appears, halo remains)
#   torch_dtype=torch.float32 end to end

Results (40 steps, seed 42, same inputs):

output_resolution	use_kv_cache	true_cfg_scale	dtype	result
512	True	1.0	bf16	correct: red scarf, photo otherwise intact
768	True	1.0	bf16	correct
1024 (default)	True	1.0	bf16	instruction ignored; haloed, over-sharpened near-copy of the input
1024	False	1.0	bf16	identical to the row above
1024	True	4.0	bf16	scarf-like red band, halo remains
1024	True	1.0	fp32	same halo (43.4 dB PSNR vs the bf16 render)

Text-to-image at 1024² and 2048² is fine with the same install; the text encoder output is unchanged between transformers 5.14.1 and 5.17.0 (bit-identical hidden states for text-only and image-conditioned prompts).

Logs

No errors or warnings are emitted; the pipeline completes normally in every configuration.

System Info

• diffusers main @ 80c7ed2 (includes Add Qwen-Image 2.1 #14804)
• transformers 5.14.1 and 5.17.0 (both tested, identical results)
• torch 2.11.0 / 2.14.0, device mps (Apple M5 Max, 128 GB), Python 3.12
• Model: Qwen/Qwen-Image-2.1 (main)

Who can help?

@yiyixuxu @sayakpaul (authors of #14804)

[sayakpaul]

@naykun

[peterc]

I can confirm on CUDA (RTX 3090 Ti) running diffusers @80c7ed2, torch 2.14.0+cu130, transformers 5.17.0, bf16 - running off of Python code driving it directly, no ComfyUI or anything like that. Slight difference though, no "haloed" copies but it replaces the subject entirely, e.g. "change the background" loses the dog or sometimes turns it into a cat. Seed makes no difference here - tried several.

It feels like the same or at least an adjacent problem as below 768 is also fine here. That said, I honed in a bit further and found the actual breaking point on mine is 896x896 (864x864 is fine) so from 52x52 latent to 54x54 latent.

[sayakpaul]

Just to confirm, do other platforms have this issue? The pipeline was contributed by the Qwen team. So, we will wait for them to have a look.

Out of curiosity, does changing prompts, using actual CFG (true_cfg_scale > 1), etc. -- does any of that help or not really?

[xocialize]

Answering both questions, plus a cross-check with an independent implementation.

Other platforms / implementations. Yes. @peterc confirms on CUDA above. On my side the decisive test was ComfyUI's native Qwen-Image 2.1 path (master @ c194dd00, no diffusers code — its own block-causal segments, prefix cache and RoPE layout): it reproduces the same 1024² degradation on the same photo and prompt, 40.1 dB PSNR to the diffusers output using ComfyUI's own seed-42 noise and its fixed shift-0.69 schedule, and 40.8 dB when I inject the diffusers noise and the exact sigmas. Same for "Change the background to a sunset beach" (39.7 dB). A third implementation (a Swift/MLX port of this pipeline) agrees with diffusers to 53.6 dB at fp32. So: three implementations, two backends, same behaviour.

Worth noting that at 1024² the result is essentially noise-independent: two ComfyUI runs with different noise agree at 49.1 dB, versus 33.2 dB between two draws at 768² where the edit is correct. The output is dominated by the reference image rather than by the sampling trajectory.

Prompts / CFG. Neither helps. Two different instructions ("Make the dog wear a red scarf", "Change the background to a sunset beach") fail the same way at 1024². true_cfg_scale=4.0 with a blank negative prompt is partial: a red band appears where the scarf should be, but the halo and over-sharpening remain. Also ruled out: use_kv_cache=False, fp32 end to end, and transformers 5.14.1 vs 5.17.0 (bit-identical hidden states).

Boundary. On my fixture, 896² and 960² targets edit correctly and 1024² does not — at the 16× VAE that is 56 and 60 latents per side working, 64 failing. @peterc sees the break lower (896² broken, 864² fine, i.e. 56 vs 54 latents). So the threshold looks fixture- and platform-dependent rather than a hard cutoff, but both land in the same regime, and his symptom is a more drastic version of the same failure (subject replaced rather than preserved-but-haloed).

One probe that may help localize it: a 768² reference paired with a 1024² target still degrades — 6,400 joint tokens there versus 8,192 at 1024/1024. So the trigger tracks the target grid extent, not the joint sequence length and not the reference grid. Text-to-image at 1024² and 2048² is fine, so it is specific to the reference-conditioned regime at that target extent.

That is what makes me think the pipeline is faithfully reproducing the published recipe and the open question is for the model authors: is a 64×64-latent edit target inside the trained regime? If not, capping output_resolution for edits (960 is clean here) would be a reasonable interim default.

[peterc]

As with @xocialize above, a CFG of 4.0 had no impact on the overall problem for me. I should note that I can't confirm if my issue is exactly the same as @xocialize's but it's very similar/'adjacent' :)

I, too, did some noodling and found that it fails if, and only if, the condition image's latent grid has max(h, w) >= 56, token count. Target size (target 1024 is fine, say) and cfg/seed/steps/kv cache have no impact. (On reflection, I note this differs from what @xocialize says above.)

I ran Codex over all of this (so this is where I fall off the rails in understanding.. I am merely parroting now ;-)) and its theory is it's something to do with the "vision tower"'s training on a 48x48 patch grid (so 768x768). Make of that what you will.

Update: I switched to a simpler "blue cube"/"add a red sphere" type example and it WORKS at 1024, BTW. So that confuses things somewhat. I'm still digging, but I'll stop reporting new stuff here at this point, as I just remembered this isn't a Qwen Image 2.1 bug tracker and so this may not even be a diffusers-related problem 😂

[naykun]

Hi @peterc @xocialize, I think the main issue is that the same seed is being used for both the initial image generation and the subsequent editing. Try using different seeds for the text-to-image (t2i) and image-to-image (i2i) steps, and let's see if that resolves the problem.

[naykun]

The issue is predominantly observed at the 1024 resolution because you set the resolution to 1024x1024 during the t2i step. If the output of the edit step is also exactly 1024, it means the initial noise is completely identical, and the dynamic scheduler will compute the same sigma values. This exacerbates the problem we refer to as "noise replay".

[peterc]

I'd be interested to see if that fixes @xocialize's problem. In my case, the t2i was at 1312 and the edit seeds were different to the one used for that (as the t2i script I use randomizes seeds). I did just re-run and verify with totally different figures, though.

However, I am going to move on as I have run numerous other image editing runs successfully at 1024 since. The only ones that have failed so far are a fox/"put a red hat on" example and the dog one @xocialize provided above.. but doing it with people, objects, landscapes, all fine. I am rarely editing photos of canids, so it may be a very isolated problem.. 😅