ControlNet gave diffusion models a structural anchor

The process wall

A text prompt alone cannot fix a composition, a pose or a horizon line, since a diffusion model reads words, not layout. The paper Adding Conditional Control to Text-to-Image Diffusion Models, posted to arXiv on 10 February 2023, addresses that gap directly: it adds a way to condition a pretrained diffusion model on a spatial map, an edge drawing, a depth map, a pose skeleton, alongside the text prompt. For a maker, that reopens a step generation had removed: producing or supplying the underlying structure a scene will follow, rather than describing it and hoping.

What the documents show

The paper's method is specific: it duplicates a pretrained diffusion model's encoding layers into a locked copy, which stays frozen, and a trainable copy, which learns the new conditioning task, connecting the two with zero-initialized convolution layers so training starts without corrupting the original model's learned weights. The authors report the approach training effectively on datasets ranging from under 50,000 to over 1 million images, and working with single or multiple conditions, with or without a text prompt. The project repository, released under an Apache-2.0 licence, lists the practical conditioning types this supports: Canny edges, depth maps, human pose, semantic segmentation, HED boundaries, M-LSD lines, scribbles and normal maps. A distributed model card for one variant, sd-controlnet-canny, confirms it pairs specifically with Stable Diffusion v1-5 and is released under the CreativeML OpenRAIL-M licence rather than Apache-2.0, since it packages the base model's own terms.

Authorship and limitation

The mechanism matters for a straightforward reason: freezing the original model and training only a copy lets ControlNet add a new capability without the enormous dataset the base diffusion model required, the paper's smallest reported training set is under 50,000 images. That is a genuine re-entry point for drawing and photographic skill: a maker's own sketch, depth pass or pose reference becomes direct input to the generation, not just a mood board. It is not a return to manual authorship of the final pixels, since the diffusion model still fills in texture and detail the conditioning map does not specify, and a supplied pose or edge map constrains composition without constraining style or content within it.

What survives

What a maker can verify is which conditioning type was used and which base model it paired with, since the repository and downstream model cards document that pairing explicitly. A workflow note that says only ControlNet was used omits the map that did the structural work.

• Check which conditioning type, edge, depth, pose or other, a described workflow used before evaluating what was actually controlled.
• Read the paper's account of locked versus trainable copies before assuming any fine-tuning method risks the base model.
• Compare the licence on a specific released variant, such as sd-controlnet-canny, against the Apache-2.0 licence on the method's own repository.

ControlNet is best described as a conditioning technique bolted onto an existing model, not a new generator: its documented contribution is making structural input cheap to add, a craft decision a maker now makes explicitly rather than one generation used to make invisibly.