Threat Model

The first step is cataloging what must be protected. For an AI system, this extends beyond traditional IT assets to include:

• Model Intellectual Property: The trained model weights, architecture, and proprietary training algorithms.
• Training & Inference Data: The datasets used for development and the live data processed during operation.
• Model Integrity & Performance: The correct, intended behavior and output quality of the system.
• Supporting Infrastructure: The APIs, vector databases, and orchestration platforms that enable the AI service.
• Sensitive Outputs: Any private information that could be reconstructed or inferred from model responses.

This component defines who might attack the system and what they can do. Adversaries are profiled by their goals, knowledge, and access level:

• Insider Threats: Employees or contractors with privileged access to training pipelines or model repositories.
• Competitors: Entities seeking to steal model functionality via model extraction or infer proprietary data.
• Malicious Users: End-users attempting prompt injection, jailbreaking, or data poisoning through API inputs.
• Advanced Persistent Threats (APTs): Sophisticated actors capable of multi-stage attacks, including supply chain compromises. Capabilities are rated based on access (white-box, gray-box, black-box) and resources, defining the attack surface.

This maps the specific techniques, tactics, and procedures (TTPs) adversaries use against each system component. Key vectors for AI systems include:

• Data Poisoning: Injecting corrupted samples into the training dataset to create backdoor attacks or degrade performance.
• Evasion Attacks (Adversarial Examples): Crafting input perturbations to cause misclassification during inference.
• Model Theft: Using model extraction attacks to clone a proprietary model via repeated API queries.
• Privacy Attacks: Executing membership inference or model inversion attacks to reveal sensitive training data.
• Prompt Injection: For LLM-based systems, crafting inputs that override system instructions to cause data exfiltration or unauthorized actions.

A technical diagram or description that traces how data moves through the AI pipeline, highlighting where trust transitions occur. Critical points include:

• Ingestion Boundaries: Where external or user-provided data enters the system (a prime location for poisoning or injection).
• Training/Finetuning Pipeline: Where the model parameters are updated, vulnerable to compromise.
• Inference Endpoint: The public-facing API where evasion and extraction attacks are launched.
• Feedback Loops: Systems where model outputs or user feedback are used for continuous learning, creating a potential pathway for iterative poisoning. Identifying these boundaries is essential for placing effective security controls.

Each identified threat is evaluated based on its likelihood and potential business impact. Impact categories are specific to AI:

• Performance Degradation: Reduced accuracy or increased latency affecting user trust and utility.
• Financial Loss: Theft of IP, cost of incident response, or regulatory fines from privacy breaches.
• Reputational Harm: Publicized model failure, biased outputs, or successful jailbreaking causing brand damage.
• Safety & Security Failures: For embodied or critical systems, physical damage or security bypasses caused by adversarial control. This assessment prioritizes mitigation efforts on high-likelihood, high-impact threats.

The final component defines the defensive measures corresponding to each high-priority threat. These are layered defenses:

• Proactive Defense: Adversarial training, red teaming, and formal verification for certified robustness.
• Data Security: Implementing differential privacy during training and rigorous data provenance checks.
• Model Security: Applying model watermarking for ownership tracking and runtime out-of-distribution detection.
• Operational Security: Implementing strict access controls, secure model deployment pipelines, and comprehensive logging for agentic observability.
• Privacy Protections: Using secure aggregation in federated learning or homomorphic encryption for encrypted inference.

An adversarial attack is a deliberate attempt to cause a machine learning model to make a mistake by feeding it specially crafted, often imperceptibly altered, input data. These attacks are a primary threat vector identified in threat modeling.

• Goal: Cause misclassification or degraded performance.
• Methods: Include Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD).
• Example: Adding subtle pixel noise to a stop sign image to make an autonomous vehicle's vision system classify it as a speed limit sign.

Data poisoning is an attack on a machine learning model where an adversary injects malicious, mislabeled, or corrupted data into the training set to compromise the model's performance or integrity. This targets the training pipeline, a critical asset in a threat model.

• Impact: Can create backdoors, reduce overall accuracy, or introduce bias.
• Defense: Requires robust data validation, provenance tracking, and anomaly detection in training data.

Model extraction, or model stealing, is an attack where an adversary uses repeated queries to a target model's API to reconstruct a functionally equivalent or similar model without authorized access to its architecture or parameters. This threatens intellectual property and can enable further attacks.

• Techniques: Use query-output pairs to train a surrogate model.
• Countermeasure: Rate limiting, output obfuscation, and monitoring for unusual query patterns.

Prompt injection is a security exploit where malicious user input is crafted to hijack the instructions of a language model's system prompt, overriding its intended behavior. This is a critical threat for agentic systems and LLM applications.

• Result: Can lead to data exfiltration, unauthorized actions, or jailbreaking.
• Defense: Requires input sanitization, privilege separation for tool calls, and rigorous prompt hardening.

A membership inference attack is a privacy attack where an adversary determines whether a specific data record was part of a model's training dataset by querying the model and analyzing its responses. This threatens data privacy and compliance.

• Mechanism: Exploits differences in a model's confidence on seen vs. unseen data.
• Mitigation: Techniques like differential privacy during training or confidence score masking.

In AI security, red teaming is the proactive practice of simulating adversarial attacks against a machine learning system to identify vulnerabilities, test defenses, and improve its overall security posture. It is the operational execution of a threat model.

• Process: Involves creating attack scenarios (e.g., prompt injections, evasion attacks) to stress-test defenses.
• Outcome: Provides empirical evidence to refine the threat model and prioritize security controls.