A 2023 paper extracted training photographs from diffusion models

The process wall

On 30 January 2023 a team spanning Google, DeepMind, ETH Zurich, Princeton and UC Berkeley published a paper testing whether diffusion generators keep a copy of what they were shown. Against Stable Diffusion, the authors selected the 350,000 most-duplicated captions in its training set and generated 500 candidate images for each, 175 million generations in total, then ranked outputs by how tightly clusters of them matched a single training photograph. That is the wall a maker meets once a prompt lands on a heavily repeated subject: the model does not always compose something new, it can retrieve a specific prior image close to pixel for pixel.

What the documents show

The paper's own count is precise. Of the 175 million generations, 94 met its strict definition of near-duplication, and manual review of the top 1,000 candidates found 13 more, for 109 total; a stricter, zero-false-positive reading still finds 50. Memorisation appeared almost only once a training image had been duplicated at least 100 times, and even then a duplicate at that level is still one training example in a million, which the authors say shows duplication, not scale, drives the effect. A separate method against Imagen, using known captions, extracted 'over a hundred near-identical replicas'. The authors' own GAN baselines memorised substantially less at matched image quality, supporting the abstract's claim that diffusion models are 'much less private than prior generative models such as GANs.' Stable Diffusion v1.4's own model card states that 'no additional measures were used to deduplicate the dataset' and that 'we observe some degree of memorization for images that are duplicated in the training data,' corroborating the mechanism from the developer's own side.

Authorship and limitation

The mechanism is duplication-driven retrieval, not a claim every output is copied. Extraction required an adversary who already suspected a duplicate existed and could query the model hundreds of times per prompt; it says nothing about an ordinary generation from an unduplicated prompt. Where it bears on authorship is what the extracted images turned out to be: 58% photographs with a recognisable person as main subject, 17% product images, 14% logos or posters; 35% carried an explicit non-permissive copyright notice, 61% none visible. A maker cannot infer novelty from a prompt's silence; the risk sits in the training data's duplication pattern.

What survives

The paper is a red-teaming study, shared with the affected developers before publication, not a registry of unsafe images. What a maker keeps is narrower and editorial: treat outputs resembling widely reposted stock photography, portraits or logos with more scepticism than a scene assembled from unrelated concepts, since that is the kind of image the paper found most exposed.

• Read the paper's figure comparing extraction rate against duplicate count.
• Compare an unexpectedly literal output against reverse-image search results before assuming it is original.
• Ask a model provider whether deduplication was applied, since the paper ties memorisation to duplicate counts.

None of this proves a given picture is stolen; it establishes that under stated, disclosed conditions, a measurable share of a diffusion model's most duplicated training images can resurface almost unchanged.