Fréchet Inception Distance (FID)

FID is based on the Fréchet distance (also called the Wasserstein-2 distance) between two multivariate Gaussian distributions. It assumes the features extracted from a set of images (real and generated) can be modeled by such distributions. The score is calculated as:

FID = ||μ_r - μ_g||² + Tr(Σ_r + Σ_g - 2(Σ_r Σ_g)^(1/2))

Where μ and Σ are the mean and covariance matrices of the real (r) and generated (g) feature distributions. A lower FID indicates the distributions are closer, meaning the generated images are more realistic.

The metric uses a pre-trained Inception-v3 network (trained on ImageNet) as a fixed feature extractor. Images are fed into the network, and activations from a specific intermediate layer (typically the last pooling layer before the final classification head) are used as a high-level feature vector. This layer captures semantic information about objects and scenes. The choice of this specific network and layer creates a standardized, pre-trained "perception" model, making FID scores comparable across different studies.

Unlike simpler metrics like Inception Score (IS), which only evaluates generated images, FID directly compares the distribution of generated images to the distribution of real images. This allows it to capture two critical failure modes of generative models:

• Mode Collapse: When a model produces low-diversity outputs (e.g., the same image repeatedly). This results in a generated distribution with low variance, leading to a high FID.
• Poor Quality: When generated images are clearly unrealistic or blurry. This shifts the mean of the generated feature distribution away from the real mean, also increasing FID.

In robotics and sim-to-real transfer, FID is repurposed to measure the visual reality gap. Instead of comparing real images to AI-generated images, it compares:

• Real-world camera feeds from a physical robot.
• Rendered images from a simulation.

A low FID score between these two domains suggests the simulation's visual rendering is photorealistic enough that features learned from simulation (e.g., for a vision-based policy) may transfer effectively to the real world. It is a proxy for visual domain shift.

Key Advantages:

• Correlates with human judgment: Lower FID scores generally align with human perceptions of image quality and diversity better than earlier metrics.
• Sensitive to both quality and diversity: Captures a more complete picture of generative performance.
• Efficient to compute: Requires only a forward pass through a pre-trained network and basic statistics.

It has become the de facto standard metric for benchmarking generative adversarial networks (GANs), diffusion models, and other image synthesis models in academic literature.

Important Limitations:

• Dataset Size Dependent: Requires a sufficiently large sample of images (often thousands) from both sets to reliably estimate the mean and covariance.
• Inception Network Bias: Inherits the biases and perceptual priorities of the ImageNet-trained Inception-v3 model. It may not align with human perception for domains far from natural images.
• Insensitive to Intra-class Diversity: Cannot distinguish between memorization of the training set and true, novel generalization.
• Not a Per-Image Metric: Provides a single score for entire datasets, not feedback on individual images.
• Assumes Gaussianity: The assumption that features follow a multivariate Gaussian is a simplification that may not hold.

While powerful, FID has known limitations. It is almost never used in isolation and is part of a broader evaluation suite.

• Insensitivity to Memorization: FID can be low even if a model simply memorizes and re-outputs training data. It must be paired with metrics like Precision and Recall for distributions to detect this.
• Feature Space Bias: It relies on features from a specific Inception network trained on ImageNet, which may not be optimal for all domains (e.g., medical images, satellite imagery). Domain-specific feature extractors are sometimes used.
• No Per-Image Score: FID is a dataset-level metric. It is complemented by per-image metrics like Inception Score (IS) or human evaluation studies (e.g., Mean Opinion Score).
• Computational Cost: Requires a large sample size (often 50k images) for a stable estimate, making it expensive for frequent evaluation during early prototyping.

The Sim-to-Real Gap, also known as the reality gap, is the performance degradation observed when a machine learning policy trained in a simulated environment is deployed on a physical system. This gap is caused by discrepancies between the simulation's modeled physics, sensor noise, visual textures, and actuator dynamics and their real-world counterparts. FID is one metric used to quantify the visual component of this gap by comparing the distribution of rendered simulation images to real camera feeds.

Domain Adaptation is a machine learning technique that aims to improve a model's performance on a target domain (e.g., the real world) by leveraging knowledge learned from a related but different source domain (e.g., a simulation). While FID measures the distance between domains, domain adaptation algorithms actively work to minimize it. Techniques include:

• Feature alignment to make source and target features indistinguishable.
• Fine-tuning models on limited real-world data.
• Using adversarial training to learn domain-invariant representations.

Distribution Shift is a change in the statistical distribution of input data or environmental conditions between the training phase (in simulation) and the deployment phase (in reality). It is the fundamental statistical cause of the sim-to-real gap. FID directly measures a form of distribution shift for images. Types of shift relevant to robotics include:

• Covariate Shift: Change in the distribution of input features (e.g., lighting, textures).
• Dynamics Shift: Change in the underlying system dynamics (e.g., friction, motor response).
• Label Shift: Change in the distribution of output labels, though less common in reinforcement learning.

Out-of-Distribution (OOD) Generalization is the ability of a machine learning model to perform accurately on input data that differs significantly from the statistical distribution of its training data. In sim-to-real, the real world is an OOD test set. A low FID score suggests the simulation's visual distribution is closer to reality, which should, in principle, support better OOD generalization for vision-based policies. Achieving OOD generalization often requires techniques like domain randomization during training to expose the model to a broader distribution of conditions.

The Inception Score (IS) is a predecessor and common point of comparison to FID for evaluating generative models. It assesses generated image quality and diversity using a pre-trained Inception network, but does so without comparing to real data. IS calculates the KL-divergence between the conditional label distribution (is the image recognizable?) and the marginal label distribution (are outputs diverse?). A key limitation is that a model can achieve a high IS by generating diverse, recognizable—but unrealistic—images, which FID corrects by incorporating the real data distribution.

The Kernel Inception Distance (KID) is a metric closely related to FID, designed to address some of its statistical limitations. While FID assumes the extracted features follow a multivariate Gaussian distribution, KID uses the squared Maximum Mean Discrepancy (MMD) with a polynomial kernel. This makes KID unbiased (its expected value is zero when distributions are identical) and often more reliable for smaller sample sizes. Like FID, it compares features from real and generated images extracted by a pre-trained Inception network.