Multimodal Adversarial Example

A single physical or digital perturbation designed to simultaneously fool multiple sensor modalities. The attacker optimizes a unified distortion that causes misclassification in each modality independently or exploits the fusion mechanism's inability to reconcile conflicting inputs.

• Example: A 3D-printed object that appears as a 'stop sign' to a camera but generates 'speed limit 80' when its surface texture is processed by a LiDAR depth estimator
• Mechanism: Joint optimization over all modality-specific loss functions, often using Expectation Over Transformation (EOT) to maintain robustness across varying physical conditions
• Key challenge: Balancing perturbation effectiveness across modalities with different dimensionalities and sensitivity profiles

Attacks that target the cross-modal attention mechanisms and feature concatenation layers where unimodal representations are combined. Rather than fooling each sensor independently, these perturbations create subtle inconsistencies that cascade into catastrophic fusion failures.

• Late fusion attacks: Perturb one modality to dominate the fused representation, suppressing correct signals from other sensors
• Early fusion attacks: Inject adversarial patterns that propagate through shared convolutional backbones before modality-specific branches diverge
• Attention hijacking: Craft inputs that cause cross-attention layers to attend to attacker-controlled regions, redirecting the model's focus away from task-relevant features

A class of attacks that deliberately creates semantic contradictions between modalities to trigger undefined behavior in the fusion logic. The model receives conflicting evidence from different sensors and resolves the ambiguity in favor of the attacker's desired output.

• Audio-visual mismatch: A video of a person saying 'yes' paired with adversarially perturbed audio that the speech recognizer transcribes as 'no', causing the fused model to default to the corrupted modality
• Depth-RGB conflict: A flat poster with adversarial texture that a monocular depth estimator interprets as a 3D obstacle, while the RGB classifier correctly identifies it as a wall
• Temporal desynchronization: Introducing micro-delays or frame drops in one modality stream to break the temporal alignment assumptions of the fusion architecture

The property that a multimodal adversarial example can be manufactured and deployed in the physical environment to fool embodied agents. This requires the perturbation to survive domain shifts including lighting changes, viewpoint variation, sensor noise, and material fabrication constraints.

• Robust physical patches: Adversarial stickers or textures optimized jointly for camera, LiDAR, and radar invisibility
• 3D-printed adversarial objects: Objects with geometry and surface properties designed to simultaneously spoof depth sensors and visual classifiers
• Acoustic-physical coupling: Perturbations that exploit the physical interaction between sound and objects, such as vibrating surfaces that confuse both microphones and laser microphones
• Fabrication constraints: The optimization must respect real-world material properties, printer resolutions, and color gamuts

The phenomenon where a multimodal adversarial example crafted against one model architecture successfully fools different, independently trained models. This property is amplified in multimodal systems due to shared pretrained backbones and common fusion paradigms.

• Cross-encoder transfer: Perturbations optimized on a ViT-based vision encoder transfer effectively to CNN-based encoders when the audio or text modality provides a consistent adversarial signal
• Fusion-agnostic attacks: Examples that succeed regardless of whether the target uses early fusion, late fusion, or transformer-based cross-attention
• Black-box amplification: The additional modality dimensions provide more degrees of freedom for the perturbation, increasing the likelihood that it lands in a region of the input space where multiple models share vulnerable decision boundaries

The fundamental imbalance where defending multimodal systems is exponentially harder than attacking them. Each added modality introduces new perturbation channels that must be individually hardened, while the attacker only needs to find the weakest cross-modal link.

• Modality-specific adversarial training must be applied to each sensor stream independently and jointly, dramatically increasing training cost
• Certified defenses like randomized smoothing must be extended to multimodal inputs, where the certification radius shrinks with each additional modality dimension
• Detection challenges: Anomaly detectors must monitor all modality streams and their fusion products simultaneously, creating a combinatorial monitoring problem
• Current state: No production multimodal system has demonstrated comprehensive adversarial robustness against adaptive, white-box attacks targeting the full sensor suite

An input to a machine learning model that has been intentionally perturbed in a way imperceptible to humans, causing the model to make an incorrect classification or decision with high confidence. This is the foundational concept upon which all adversarial attacks are built.

• Imperceptibility: Perturbations are constrained by an Lp-norm budget (e.g., L∞ ≤ 8/255)
• Targeted vs. Untargeted: Can force a specific wrong class or just any misclassification
• Transferability: Examples crafted on one model often fool others with different architectures

An attack that creates adversarial perturbations in the physical world—such as stickers, 3D-printed objects, or altered textures—designed to fool perception systems under varying lighting, angles, and distances.

• Robustness to Transformations: Must survive viewpoint shifts, occlusions, and sensor noise
• Expectation Over Transformation (EOT): Optimizes perturbations over a distribution of real-world conditions
• Real-world examples: Stop sign stickers that cause misclassification, adversarial glasses that defeat facial recognition

A defensive technique that augments the training dataset with adversarially perturbed examples labeled with the ground-truth class, forcing the model to learn robust decision boundaries.

• Min-Max Formulation: Inner maximization finds worst-case perturbation; outer minimization reduces loss
• Projected Gradient Descent (PGD): The standard attack used to generate training adversaries
• Trade-off: Improves robustness but often reduces accuracy on clean data
• Multimodal extension: Requires crafting perturbations that span all input modalities simultaneously

An attack on an embodied agent that injects falsified data into hardware sensors such as LiDAR, cameras, or inertial measurement units (IMUs) to corrupt the agent's perception of its physical environment.

• LiDAR Spoofing: Projects timed laser pulses to inject phantom points or delete real obstacles from 3D point clouds
• GPS Spoofing: Overrides satellite signals to feed false position coordinates
• Cross-modal implications: Spoofing one sensor can create inconsistencies that cascade through sensor fusion pipelines

A formal, provable guarantee that a model's prediction will not change for any input perturbation within a specified Lp-norm bound.

• Randomized Smoothing: Constructs a smoothed classifier by adding Gaussian noise and aggregating predictions, providing a provable L2 radius
• Interval Bound Propagation (IBP): Propagates input bounds through the network to certify output stability
• Limitation: Certified radii are often smaller than empirical robustness; multimodal certification remains an open research problem

An attack methodology that assumes full knowledge of a defense mechanism and is specifically designed to circumvent it, representing the most rigorous evaluation standard for adversarial robustness.

• Defeats Gradient Masking: Identifies and bypasses obfuscated gradients that give false security
• White-box assumption: Attacker knows architecture, parameters, and defense details
• Evaluation imperative: Any defense not tested against adaptive attacks cannot be considered reliable
• Multimodal context: Adaptive attackers exploit the weakest modality or the fusion mechanism itself