Attack Success Rate (ASR)

ASR is calculated as the ratio of successful adversarial attempts to total attempts. A successful attack is strictly defined as a model output that matches the attacker's harmful intent, bypassing safety filters.

• Formula: ASR = (Number of Harmful Outputs / Total Attack Queries) × 100%
• Granularity: Can be measured per attack technique (e.g., GCG ASR vs. TAP ASR) or aggregated across a full red teaming suite.
• Thresholds: An ASR > 50% indicates a critical vulnerability; ASR < 1% is the target for production-grade guardrails.

ASR depends on a robust taxonomy of harmful content. Automated classifiers evaluate model responses against predefined safety policies to determine if an attack succeeded.

• Policy Categories: Hate speech, violence, self-harm, illegal advice, PII leakage, and sexual content.
• Classifier Types: Fine-tuned BERT-based models, LLM-as-a-judge evaluators, and keyword heuristics.
• False Positives: Overly sensitive classifiers inflate ASR; rigorous human alignment of the evaluator is critical for metric validity.

ASR must be contextualized by the attack's interaction model. Multi-turn attacks like Crescendo often achieve higher ASR than single-turn prompts.

• Single-Turn ASR: Measures immediate jailbreak success from one prompt.
• Multi-Turn ASR: Tracks success over a conversation, where benign-seeming dialogue gradually erodes safety guardrails.
• Metric Distinction: A model may have 0% single-turn ASR but 40% multi-turn ASR, revealing a hidden vulnerability surface.

ASR is inversely correlated with the adversarial budget—the number of queries or compute resources an attacker expends. Efficient attacks achieve high ASR with minimal queries.

• Query-Limited ASR: Measures success within a fixed budget (e.g., 100 queries).
• Unlimited Budget ASR: Theoretical maximum success rate with infinite retries.
• Optimization Goal: Automated red teaming tools like GCG optimize for high ASR with low query counts, simulating realistic attacker constraints.

A critical ASR dimension is transferability—whether an adversarial prompt or suffix generated against one model succeeds against a different, black-box target.

• Transfer ASR: The success rate when applying attacks optimized on a surrogate model (e.g., Llama) to a target model (e.g., GPT-4).
• Universal Triggers: Inputs with high transfer ASR across diverse architectures indicate systemic vulnerabilities in alignment training.
• Defense Implication: Low transfer ASR suggests model-specific weaknesses; high transfer ASR signals a fundamental safety flaw.

ASR is not static. Continuous Automated Red Teaming (CART) tracks ASR over time to detect safety regressions caused by model updates or data drift.

• Regression Alerts: A spike in ASR after a fine-tuning run indicates the update inadvertently weakened safety alignment.
• Drift Correlation: ASR increases may correlate with shifts in production input distributions, requiring recalibration of guardrails.
• Dashboarding: Security teams monitor ASR trends alongside standard performance metrics in CI/CD pipelines.