Simulation Policy Extraction

The attack requires no internal knowledge of the model architecture. The adversary operates purely as an oracle, submitting state-action pairs and observing the policy's output distribution.

• Exploits standard prediction APIs or agent interfaces
• Collects (state, action) tuples to build a distillation dataset
• Does not require gradients, weights, or architecture details
• Effective against policies deployed behind firewalled endpoints

The extracted clone does not need to be a bitwise copy. It must only achieve functional equivalence — producing the same action distribution for any given state within the operational domain.

• The clone policy maximizes agreement on output actions
• Often uses a student model with a different architecture than the victim
• Agreement rates exceeding 99% on held-out state distributions are achievable
• Enables downstream adversarial analysis offline

The attacker systematically probes the policy's input manifold to maximize coverage of the state-action mapping. Strategic query selection minimizes the number of queries needed for high-fidelity extraction.

• Uses active learning to select maximally informative queries
• Targets decision boundaries where small state changes flip the action
• Exploits simulation parameterization to generate diverse synthetic states
• Query budgets can be as low as 10^4 to 10^6 samples for complex policies

The extracted clone exposes the victim's trade secrets — optimized control strategies, edge-case handling logic, and competitive decision heuristics developed through expensive simulation training.

• Reveals the reward function's implicit priorities
• Exposes brittle failure modes and exploitable blind spots
• Enables competitors to replicate years of RL training in days
• Particularly damaging for reinforcement learning policies in robotics and trading

Once extracted, the clone becomes a surrogate model for offline adversarial attack development. Attackers can craft input perturbations that transfer to the victim policy with high probability.

• Enables white-box attack techniques against the extracted clone
• Transferability rates of 80-95% are common between clone and victim
• Facilitates discovery of catastrophic failure states without alerting the victim
• Accelerates development of evasion and manipulation exploits

Policies trained in simulation are uniquely vulnerable because the simulator itself can be used to generate unlimited, perfectly labeled training data for the extraction process.

• The attacker can instantiate their own simulation instance
• Uses the victim policy as an oracle to label states from the attacker's simulator
• No real-world data collection costs or physical constraints
• Enables unbounded extraction without rate limiting or physical wear

The process of discovering and exploiting unintended loopholes in a reinforcement learning reward function to achieve high scores without completing the intended task. This is a precursor to policy extraction, as a hacked reward signal reveals the objective structure.

• Specification Gaming: An agent satisfies the literal reward criteria while violating the designer's intent
• Reward Tampering: Directly manipulating the reward signal in the simulation loop
• Proxy Reward Discovery: Reverse-engineering the true objective from observed agent behavior

An adversarial technique that identifies and leverages discrepancies between simulation and reality to cause a policy trained in simulation to fail upon deployment. Policy extraction often serves as a reconnaissance step for this attack.

• Dynamics Mismatch: Differences in physics, friction, or contact modeling
• Rendering Gap: Visual discrepancies between synthetic and real sensor data
• Latency Exploitation: Targeting timing assumptions that hold in simulation but not in hardware

A data poisoning technique targeting the domain adaptation module that maps simulated features to real-world features. By corrupting this bridge, an attacker causes the transferred policy to behave incorrectly in deployment.

• Adversarial Adaptation: Injecting malicious paired examples into the adaptation training set
• Feature Collapse: Forcing the adaptation layer to map diverse real inputs to a narrow simulated representation
• Transfer Backdoor: Embedding a trigger that activates only after domain adaptation

The unauthorized creation of a functional copy of a digital twin environment. This cloned simulation enables adversaries to run unlimited policy extraction queries offline without rate limiting or detection.

• Asset Exfiltration: Stealing 3D models, textures, and physics parameters
• Offline Brute-Forcing: Running millions of extraction queries in the cloned environment
• Vulnerability Research: Identifying exploits in the twin before attacking the physical system

A technique where an attacker uses observed behavior to reverse-engineer the kinematic constraints of a simulated robot. Once the joint limits, link lengths, and degrees of freedom are known, the attacker crafts inputs that force the agent into singular or unstable configurations.

• Jacobian Null-Space Exploitation: Identifying configurations where certain motions become impossible
• Singularity Forcing: Driving the robot into a pose where it loses a degree of freedom
• Workspace Boundary Mapping: Probing to discover the exact reachable volume of the manipulator