A distance score ranks generators without judging one image

The process wall

A maker comparing two generators is rarely shown a single number in the tools themselves, but papers behind those tools almost always report one: the Fréchet Inception Distance. The paper that introduced it, submitted 26 June 2017 by Heusel and colleagues, was not about text-to-image models at all; it proposed a training-rate fix for GANs and, to measure whether that fix worked, needed a better score than the existing Inception Score. A metric built to compare two GANs' output distributions during training became a default yardstick for comparing today's generators, well outside its original purpose.

What the documents show

The paper's own method is a comparison of statistics, not of pictures: it passes real and generated images through a pretrained Inception network, fits a multivariate Gaussian to each set's feature activations by mean and covariance, and reports the Fréchet distance between those two Gaussians. It shows this distance rising smoothly as images are corrupted by noise, blur or swirling, offered as evidence the score tracks perceptible quality loss. It says nothing about any single image, only how similar two whole distributions are. Years later, separate teams tried to close that gap with human judgment: ImageReward, submitted 12 April 2023, trained a reward model on 137,000 expert comparisons and reports it 'outperforms existing scoring models and metrics' at predicting which of two images a person prefers; Pick-a-Pic, submitted three weeks later, built an open dataset from real users' own preference choices and reports its PickScore 'correlates better with human rankings' than prior automatic metrics.

Authorship and limitation

The mechanism to name is the gap between a distributional statistic and a preference model. FID only says whether a batch of outputs resembles a reference batch in aggregate feature space; it cannot say a specific image is well composed, on-brief or usable, and a model can improve its FID by resembling the reference set's average rather than by being useful to a maker. ImageReward and Pick-a-Pic replace that comparison with learned human preference, but each is bounded by whose preferences it learned, paid expert annotators in one case, self-selected app users in the other, so a high score reflects that population's taste, not an objective standard.

What survives

What survives for a maker is caution about leaderboard numbers: a lower FID or higher preference score describes a model's standing against a particular reference set or rater pool, and switching either changes the ranking. A model card's benchmark numbers are worth reading for which dataset and raters produced them before treating the number as a quality judgment.

• Check which reference dataset a reported FID score was computed against.
• Read whether a preference score comes from expert annotators or app users, since ImageReward and Pick-a-Pic used different populations.
• Compare a model's benchmark ranking against your own judgment on a brief-specific task, not a generic prompt set.

None of these scores were built to replace a maker's own judgment of a particular image for a particular brief; each was built to compare models to each other under a stated, narrower definition of similarity or preference.