Images, video, voice and music
ControlNet
In short
ControlNet is a neural-network add-on that gives a text-to-image diffusion model extra spatial guidance, such as an edge map, depth map or human pose, so the generated picture follows a chosen layout1.
- Also called
- Control Net
- Topic
- Images, video, voice and music
- Level
- Some background helps
- First described
- Lvmin Zhang, Anyi Rao and Maneesh Agrawala introduced ControlNet in ‘Adding Conditional Control to Text-to-Image Diffusion Models’, posted in February 20231.
A normal text-to-image model mainly follows words. ControlNet adds another input that describes where important shapes should go. The original paper demonstrates edge, depth, segmentation and human-pose conditions with Stable Diffusion1.
The diffusion model itself creates an image through a sequence of denoising steps, and text or other inputs can guide that process2.
It works by keeping the original, pretrained model locked and making a trainable copy of parts of its network. Zero-initialised convolution layers connect the copy to the original, so the added branch starts by making no change and learns the condition during training1.
ControlNet is not a prompt, a new image format or a guarantee that every pixel will be copied. It is a conditioning architecture for diffusion models. The result still depends on the text prompt, the model, the condition image and generation settings1.
In an image tool, a user might upload a rough pose drawing or an edge map, choose a ControlNet condition, and add a text description. The system can then generate a new scene that keeps the supplied spatial structure while changing its subject or style1.
Example
You photograph a person standing with one arm raised, extract a pose map, and ask for ‘a paper-cutout astronaut’. ControlNet uses the pose as spatial guidance while the diffusion model supplies the astronaut’s appearance1.
Why it matters
ControlNet makes image generation more controllable than words alone. It is useful when pose, composition or outlines matter as much as the subject description1.
Questions people ask
What does ControlNet control?
It can condition image generation on spatial information such as edges, depth, segmentation or human pose. The original paper also describes using one or multiple conditions with Stable Diffusion1.
Is ControlNet a model or a plugin?
It is a neural-network architecture that adds conditional control to a pretrained text-to-image diffusion model. Some applications present a trained ControlNet as a selectable add-on, but the underlying idea is an architecture1.
Does ControlNet copy an image exactly?
No. It uses an input condition such as a pose or edge map to guide spatial structure. The diffusion model still generates the visual content from the condition, prompt and its learned weights1.