alibaba-pai/Qwen-Image-2.1-Fun-Controlnet-Union

---

license: other license_link: LICENSE license_name: qwen-research library_name: videox_fun tags:

• controlnet
• controlnet-union
• text-to-image
• image-to-image
• image-inpainting tasks:
• text-to-image-synthesis

---

Qwen-Image-2.1-Fun-Controlnet-Union

Overview

Qwen-Image-2.1-Fun-Controlnet-Union is a ControlNet-Union branch for Qwen-Image 2.1 (the flow-matching text-to-image DiT). A single checkpoint drives 8 structural control conditions (Canny, Depth, Grayscale, HED, Lineart, MLSD, Pose, Scribble) and image inpainting, without per-condition weights. The checkpoint holds only the control branch (control_img_in plus 16 control_blocks, about 7.0 GB) and is loaded on top of the base Qwen-Image 2.1 transformer.

Model Card

Name	Description
Qwen-Image-2.1-Fun-Controlnet-Union.safetensors	ControlNet-Union branch weights for Qwen-Image 2.1. Contains only the control branch (control_img_in + 16 control_blocks, about 7.0 GB); loaded with strict=False on top of the base Qwen-Image 2.1 transformer. One checkpoint covers 8 control conditions and image inpainting.

Model Features

• Union control over 8 conditions: one checkpoint handles Canny, Depth, Grayscale, HED, Lineart, MLSD, Pose and Scribble control images for text-to-image generation — no per-condition checkpoint switching.
• Dense control injection: the control branch attaches a skip to every 2nd of the 32 transformer blocks (control_layers = [0, 2, 4, …, 30], 16 injection points). Each control skip is added back to the main branch through zero-gated before_proj / after_proj projections, giving tight structural adherence while the base model stays frozen.
• Control and inpainting share one branch: the control input is widened to control_in_dim = 129 — control latents (64) | mask (1) | masked-image latents (64). For pure control the mask / masked-image channels are zero-padded; for inpainting the same branch re-draws the masked region from the prompt. The two can also be combined — a control image and a mask are fed together, so the re-drawn region follows both the prompt and the given structure.
• CFG-distilled fast sampling: the standalone example scripts run with guidance_scale = 1.0 (single forward pass per step, no classifier-free guidance needed).
• control_context_scale scales every control skip before it is added to the main branch: 1.0 is the strongest control (used for all results below), lower values weaken the guidance, 0.0 switches the control branch off.
• Prompt-friendly: write a prompt that describes the whole target image; the masked region is conveyed by the mask channel, not by the text. Detailed prompts give better stability.
• Qwen-Image 2.1 encodes the prompt (and any condition image) with a Qwen3-VL text encoder + processor, and its VAE decodes to RGBA, so every preview is saved as PNG.

Supported control conditions

Condition	Control signal
Canny	Canny edge map
Depth	Monocular depth map
Grayscale	Grayscale (luminance) image
HED	HED edge detection map
Lineart	Line-art extraction
MLSD	Line-segment detection map
Pose	DWPose skeleton
Scribble	Free-hand / sketch lines

Any ordinary RGB control image at the target canvas works; the model tolerates different line thickness, thresholds and crops.

Results

All samples below are generated with num_inference_steps = 40, control_context_scale = 1.0, seed 43. In each column the top row is the control image, the bottom row is the output.

See the full description on the original page