Robustness Evaluation

This method involves generating adversarial examples—inputs intentionally perturbed to cause model failure while remaining imperceptible or semantically similar to a human. The goal is to probe the model's decision boundaries and expose vulnerabilities.

• Key Techniques: Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD), and Carlini & Wagner (C&W) attacks.
• Purpose: Measures a model's susceptibility to malicious manipulation and quantifies its adversarial robustness.
• Example: Adding subtle pixel-level noise to a stop sign image that causes an autonomous vehicle's vision system to misclassify it as a speed limit sign.

This technique evaluates a model's resilience to naturally occurring noise and data corruption, simulating real-world sensor errors, transmission artifacts, or low-quality inputs.

• Common Corruptions: Gaussian noise, blur, JPEG compression artifacts, brightness/contrast shifts, and missing data (e.g., dropout).
• Benchmarks: Datasets like ImageNet-C and CIFAR-10-C provide standardized corruption severities.
• Output: Produces a corruption error rate, showing how gracefully performance degrades as input quality decreases. This is critical for models deployed in edge or noisy environments.

This method tests a model's ability to identify when an input is statistically different from its training data (out-of-distribution) and ideally, to abstain from making a high-confidence prediction.

• Core Challenge: Distinguishing between in-distribution and OOD samples without explicit labels.
• Evaluation Metrics: AUROC (Area Under the Receiver Operating Characteristic curve) and FPR@95TPR (False Positive Rate when True Positive Rate is 95%).
• Significance: Prevents models from making dangerously confident predictions on novel, unseen data types, a key safety requirement.

This approach subjects a model to a high volume of random or semi-random input variations to discover rare failure modes and edge cases not covered by deterministic tests.

• Methodology: Uses techniques like fuzzing (generating random invalid inputs) or Monte Carlo simulations with parameterized noise.
• Goal: Uncover unexpected model behaviors, memory leaks, or performance degradation under sustained anomalous load.
• Application: Essential for safety-critical systems (e.g., finance, healthcare) where the cost of a rare failure is extremely high.

This method verifies that a model's predictions are appropriately stable (invariant) or consistently transform (equivariant) under a set of predefined, semantically meaningless input transformations.

• Invariance Test: The model's output should not change for transformations that do not alter the label (e.g., a classifier's prediction should be the same for a rotated image of a cat).
• Equivariance Test: The model's output should transform in a predictable way (e.g., an object detector's bounding boxes should rotate with the image).
• Use Case: Ensures models learn the correct features and are not overly sensitive to irrelevant variations in the data.

This qualitative method employs human experts (red teams) to manually craft creative, adversarial prompts or inputs designed to 'break' the model, especially for generative AI and language models.

• Focus: Exposing jailbreaks, prompt injection vulnerabilities, biased outputs, and logical inconsistencies.
• Process: Iterative and exploratory, relying on human intuition to find failure modes automated methods miss.
• Outcome: A catalog of concrete failure cases used to harden the model via improved training data, guardrails, or system design. This is a cornerstone of LLM security evaluation.

A systematic security evaluation method that probes AI models with intentionally crafted inputs designed to cause failures, misclassifications, or unintended behaviors. This is a proactive subset of robustness evaluation focused on malicious intent.

• Purpose: Expose vulnerabilities to manipulation before deployment.
• Common Techniques: Includes gradient-based attacks (e.g., FGSM, PGD) and score-based attacks that perturb inputs within small, often imperceptible bounds.
• Example: Adding subtle pixel noise to a stop sign image to cause an autonomous vehicle's vision model to classify it as a speed limit sign.

The process of testing a model's performance on data whose statistical properties differ significantly from its training distribution. This measures generalization robustness to novel, unexpected, or edge-case inputs.

• Key Contrast: Different from adversarial testing, as OOD data is not maliciously engineered but naturally atypical.
• Common Benchmarks: Use datasets like ImageNet-C (corrupted images) or WILDS for domain shift.
• Primary Metric: The performance drop (accuracy, F1 score) between in-distribution and OOD test sets quantifies distributional robustness.

A security-inspired, human-driven evaluation practice where testers role-play as adversaries to manually generate challenging prompts or inputs that expose model failures, biases, or safety violations. It complements automated adversarial testing.

• Focus Area: Particularly critical for Large Language Models (LLMs) to uncover harmful content generation, prompt injection vulnerabilities, or jailbreaks.
• Process: Involves iterative probing, creativity, and domain expertise to find failure modes automated systems might miss.
• Output: A catalog of failure cases used to harden models via fine-tuning or guardrails.

Monitoring infrastructure that identifies when the statistical properties of live input data (data drift) or model predictions (concept drift) change over time compared to a baseline. This is a production-level robustness safeguard.

• Proactive vs. Reactive: Detects degradation as it happens, enabling retraining or mitigation before user-facing performance collapses.
• Common Techniques: Uses statistical tests (e.g., Kolmogorov-Smirnov, PSI), model-based detectors, or monitoring embedding distributions.
• Link to Robustness: A model robust to distributional shifts will trigger fewer false-positive drift alerts in production.

A quantitative measure used to audit an AI system for unfair or discriminatory outcomes across different demographic groups. Evaluating robustness across subgroups is a critical component of responsible AI.

• Core Principle: A robust model should perform equitably across protected attributes like race, gender, or age.
• Common Metrics: Disparate Impact Ratio (compares selection rates), Equal Opportunity Difference (compares true positive rates).
• Evaluation Process: Requires disaggregated evaluation on test sets containing subgroup labels to measure performance gaps.

A broad evaluation methodology that subjects a system to extreme operational conditions—such as high load, noisy data, or resource constraints—to assess its stability and failure modes. In AI, this encompasses robustness, latency, and reliability.

• AI-Specific Stressors: Includes high query load (measuring throughput degradation), inference with corrupted inputs, or operation in low-memory environments.
• Goal: Identify breaking points and ensure graceful degradation rather than catastrophic failure.
• Relationship: Stress testing for latency and throughput (P99 latency under load) is a key infrastructure complement to output robustness testing.