Researchers named a decay from training on model output

The process wall

On 24 July 2024, Nature published a paper by Shumailov, Shumaylov, Zhao, Papernot, Anderson and Gal naming a specific failure mode: successive generations of a generative model trained on the previous generation's own output. An earlier preprint version, submitted 27 May 2023 under the title 'The Curse of Recursion', had already set out the core experiments; the Nature version carries the same finding through peer review to a wider readership. The wall it names sits ahead of any specific tool: as generated content fills more of the web that future training sets are scraped from, later models increasingly train on earlier models' output rather than on human-made material.

What the documents show

The paper defines 'model collapse' as a process in which 'the tails of the original content distribution disappear' across generations of training on generated data, causing 'irreversible defects in the resulting models'. It demonstrates the effect in three kinds of generative model, Gaussian mixture models, variational autoencoders, and large language models, arguing from this spread that the failure is a general property of learned generative models rather than a quirk of one architecture. It separates two stages: 'early model collapse', where a model starts losing information about the rarer tails, and 'late model collapse', where it converges toward a distribution with 'substantially reduced variance' barely resembling the original, attributing the effect to three compounding error sources: finite-sample approximation error, limited model expressivity, and error from the training procedure itself.

Authorship and limitation

The paper's own experiments are on text and on the two named non-text architectures, Gaussian mixtures and variational autoencoders, not on image diffusion models directly; its opening line invokes 'stable diffusion' only to frame generative AI's reach, not as a tested system. Because a variational autoencoder is a component many image pipelines use to compress images before diffusion operates on them, the paper's VAE result is a documented, adjacent data point for image workflows, not a direct finding about diffusion training. That distinction matters for a maker deciding how far to extend the headline claim: the mechanism, tails of a distribution disappearing under recursive training, is general in the paper's own theoretical argument, but its measured evidence for image-specific pipelines is narrower than for language models.

What survives

What the paper argues survives is access to genuinely human-made data, which it calls increasingly valuable precisely because model-generated content is filling the same web that future training sets are scraped from. Editorially, a maker curating or licensing a dataset should treat provenance, whether an image is human-made or generated, as a property worth tracking now, on this paper's argument, rather than a concern for dataset builders alone.

• Read the paper's definitions of early versus late model collapse.
• Check whether a dataset's documentation states how it filtered or flagged synthetic content.
• Compare the preprint and Nature versions to see what the peer-review process changed.

The paper's claim is theoretical and cross-architecture rather than a measured image-generation result, which is exactly why a maker should read its own scope carefully before extending it to any specific tool.