difficulty 4/5
Fréchet Inception Distance
Fit a Gaussian to Inception features for real and generated images, then measure the distance between them. The number you compute is not the number you want.
Intuition
Generative models have no right answer to check against, so the metric has to compare distributions rather than predictions. FID does that in the crudest way that works: push both sets of images through a classifier, keep an intermediate layer, fit a Gaussian to each set of features, and measure the distance between the two Gaussians.
It is the standard number in generative vision — DDPM reports "a state-of-the-art FID score of 3.17" on CIFAR-10 [3, Abstract] and every paper since quotes one.
The difficulty is not that FID is a bad idea. It is that the quantity you compute from a finite sample is not the quantity FID is defined to be, the gap does not go away at any sample size anyone uses, and the size of the gap depends on which model you are measuring. Chong and Forsyth state the consequence bluntly: "all comparisons using FID or IS as currently computed are unreliable" [1, Abstract].
Formal definition
How it works
Where it's used
Why & when
Worked example
Pitfalls
Check yourself
3 questions
The questions on this page come with access. They are graded on the server, so the answers stay there until you have given yours.
Sources
- Effectively Unbiased FID and Inception Score and where to find them (2020)paperhttps://ar5iv.labs.arxiv.org/html/1911.07023
- Demystifying MMD GANs (2018)paperhttps://ar5iv.labs.arxiv.org/html/1801.01401
- Denoising Diffusion Probabilistic Models (2020)paperhttps://ar5iv.labs.arxiv.org/html/2006.11239