FID Score (Fréchet Inception Distance)

The FID score does not compare images pixel-by-pixel. Instead, it measures the similarity between two multivariate Gaussian distributions.

• Real Images: A distribution of features extracted from a dataset of real images.
• Generated Images: A distribution of features from the synthetic images. A lower FID indicates the two distributions are more similar, meaning the generated images are statistically closer to real ones. This makes it robust to small, perceptually irrelevant pixel differences.

FID uses a pre-trained Inception-v3 network as a fixed feature extractor. Images are passed through the network, and activations from the final pooling layer (a 2048-dimensional vector) are used as their feature representation.

• This layer captures high-level semantic features relevant to object recognition.
• Using a pre-trained, fixed network ensures the metric is consistent and task-agnostic, not biased toward any specific generative model's training.

The core calculation is the Fréchet distance (also called the Wasserstein-2 distance) between two multivariate Gaussians. The formula is: FID = ||μ_r - μ_g||² + Tr(Σ_r + Σ_g - 2(Σ_r Σ_g)^(1/2)) Where:

• μ_r, μ_g are the mean feature vectors for real and generated distributions.
• Σ_r, Σ_g are the covariance matrices.
• Tr() is the trace of a matrix. This mathematically captures both the difference in central tendency (means) and the difference in spread and structure (covariances) of the features.

FID was developed to address key shortcomings of the earlier Inception Score (IS).

• IS evaluates only generated images, measuring diversity and recognizability within the synthetic set. It does not compare to real data.
• FID directly compares to the target real distribution, making it a more reliable measure of realism and fidelity.
• FID is more sensitive to mode collapse (where a generator produces limited variety) because it measures the full covariance of the distribution.

FID is particularly effective at detecting common failure modes in generative models.

• Mode Collapse: If a generator produces only a few types of images, the covariance matrix of the generated features (Σ_g) shrinks, causing a high FID.
• Artifacts: Systematic visual flaws (e.g., blurry textures, distorted shapes) shift the mean feature vector (μ_g) away from the real distribution's mean, also increasing the score. This makes it a useful diagnostic tool during model development.

While a standard, FID has important practical constraints.

• Requires a Sufficient Sample Size: Typically, at least 10,000 images are used from each distribution for a stable estimate.
• Computationally Intensive: Calculating covariance matrices for high-dimensional features is expensive for very large sample sets.
• Feature Space Bias: It inherits any biases in the Inception-v3 network, which was trained on ImageNet. It may not optimally evaluate domains far from natural images (e.g., medical scans, abstract art).
• Single-Number Summary: A good FID score does not guarantee every individual image is high quality; it measures overall distributional fidelity.

The CLIP Score is an automatic evaluation metric that quantifies the semantic alignment between a generated image and its corresponding text prompt. It uses the cosine similarity between the image and text embeddings produced by a pre-trained CLIP model.

• Measures Prompt Fidelity: Unlike FID, which measures distributional similarity, CLIP Score directly assesses how well the generated image matches the textual description.
• Complementary to FID: Often used alongside FID to provide a more holistic evaluation (FID for realism/distribution, CLIP for text-image alignment).
• Limitations: Can be gamed by models that produce images highly aligned with CLIP's embedding space but lacking in fine-grained visual quality.

The Inception Score is an earlier metric for evaluating generative models, particularly GANs. It assesses both the quality and diversity of generated images using a pre-trained InceptionV3 classifier.

• Calculation: Measures the KL-divergence between the conditional label distribution (image quality) and the marginal label distribution (diversity). A high score indicates images are both recognizable and varied.
• Key Criticisms: It does not compare generated images to a real dataset, making it possible to achieve a high score with a model that generates diverse but unrealistic images. This flaw led to the development of FID, which directly compares distributions.
• Historical Context: Largely superseded by FID in modern research due to its lack of a reference real dataset.

Precision and Recall for Distributions is a framework that decomposes generative model performance into two separate metrics, offering more nuanced diagnostics than a single score like FID.

• Precision (Quality): Measures how much of the generated distribution falls within the support of the real data distribution. High precision means generated images are realistic.
• Recall (Diversity): Measures how much of the real data distribution is covered by the generated distribution. High recall means the model captures all modes of the real data.
• Diagnostic Power: A model can have high precision but low recall (mode collapse), or high recall but low precision (generates unrealistic images). FID conflates these two aspects into a single number.

The Kernel Inception Distance is a metric similar to FID but uses the squared Maximum Mean Discrepancy (MMD) with a polynomial kernel on Inception network features.

• Advantages over FID: KID is unbiased (its expected value is zero when comparing identical distributions) and is often considered to have better statistical properties, especially with smaller sample sizes.
• Interpretation: Like FID, a lower KID indicates greater similarity between real and generated feature distributions.
• Common Use: Frequently reported alongside FID in academic papers to provide a more robust statistical comparison.

Human Evaluation is the gold standard for assessing generative model output, involving subjective judgments from human raters on criteria that automated metrics may miss.

• Common Tasks: Includes Visual Turing Tests (can humans distinguish real from fake?), preference rankings (A/B tests), and ratings on specific attributes like aesthetic quality, realism, and prompt adherence.
• Limitations: Expensive, time-consuming, and difficult to scale. Results can be subjective and vary between raters.
• Role: Used to validate and ground-truth automated metrics like FID and CLIP Score. A good automated metric should correlate strongly with human judgment.

LPIPS is a perceptual metric that quantifies the perceived similarity between two images. It is learned from human perceptual judgments and is based on the deep features of a neural network (often a pre-trained AlexNet or VGG).

• Use in Generation: While often used for image translation and super-resolution tasks, LPIPS can measure the diversity within a set of generated images by calculating the average distance between pairs. This helps detect mode collapse.
• Difference from FID: LPIPS measures perceptual similarity between individual image pairs, whereas FID measures statistical similarity between entire datasets.
• Application: Can be used to ensure a model generates perceptually diverse outputs, complementing FID's assessment of overall distributional fidelity.