Readers rated AI poems well and could not spot them reliably

The process wall

A poet or workshop leader often assumes a trained reader can hear the difference between a poem shaped by years of a poet's practice and one produced on request by a language model. A study published in Scientific Reports on 14 November 2024 tests that assumption directly, using what the authors call a ‘human out of the loop’ method: researchers Brian Porter and Edouard Machery took ‘the first 5 poems generated’ for each poet with no cherry-picking and no feedback to the model, prompted only with ‘write a short poem in the style of …’.

What the documents show

The design used ChatGPT-3.5 to generate five poems ‘in the style of’ each of ten poets spanning Geoffrey Chaucer to a living poet, Dorothea Lasky. In the first experiment, 1,634 participants each judged ten poems — five genuine, five AI-generated — for one assigned poet; accuracy came to 46.6%, below chance, and the paper reports participants were significantly more likely to call an AI poem human-written than to correctly call a genuine poem human-written. A second experiment recruited 696 Prolific participants to rate a ten-poem subset across fourteen qualities under three framing conditions, finding AI-generated poems rated more favourably on qualities such as rhythm and beauty.

Authorship and limitation

The mechanism the authors name is a heuristic failure in readers, not a claim about literary merit: their own account is that ‘the simplicity of AI-generated poems may be easier for non-experts to understand,’ leading readers to ‘misinterpret the complexity of human poems as incoherence generated by AI.’ That is a specific, falsifiable claim about non-expert expectations under these conditions, distinct from a general statement that machine-written poetry has overtaken human poetry in quality. The below-chance accuracy figure and the preference ratings are two separate measurements the paper reports side by side, and collapsing them into one impression — readers simply prefer AI poetry — drops the more precise finding that readers could not reliably tell the difference in the first place.

What survives

The authors' own limitations note states the results are ‘limited to the most recent generation of generative language models, and to people's current beliefs and biases regarding AI-generated texts,’ and that what ‘sounds human’ may shift as models and expectations change. Editorially, a workshop or contest that relies on reader vote to screen for AI-written entries should not assume this result is stable across time or across a different model, and should specify its own detection method rather than defer to reader intuition.

• Check which model, which poets, and how outputs were selected before drawing a conclusion from any similar identification exercise.
• Note the difference between accuracy at telling AI from human and a rated preference for one over the other, since this study measured both separately.
• Revisit the paper's own stated scope before generalising past it, since the authors limit the claim to current models and current reader expectations.

The finding is dated and model-specific by the authors' own account, a measurement of one moment in reader expectation rather than a permanent verdict on machine versus human poetry.