Systemic Risk Threshold

The most quantifiable systemic risk threshold is measured in floating-point operations (FLOPs) used during training. The EU AI Act initially proposed a threshold of 10^25 FLOPs, a compute budget that only the largest frontier models exceed. This metric is favored by regulators because it is objective, verifiable, and difficult to manipulate. Training runs that cross this boundary automatically trigger mandatory reporting, adversarial red-teaming, and incident disclosure requirements. The threshold is designed to be updated dynamically as hardware efficiency improves.

Beyond raw compute, regulators define thresholds based on dangerous capabilities that emerge unpredictably. These include:

• Chemical/Biological Design: The ability to synthesize novel compounds or pathogens
• Autonomous Replication: The capacity to self-improve or copy itself without human intervention
• Strategic Deception: The ability to manipulate, persuade, or deceive humans at expert levels
• Cyber Offense: Proficiency in discovering and exploiting zero-day vulnerabilities Models exhibiting these capabilities trigger additional oversight regardless of training compute.

Leading AI labs implement internal Responsible Scaling Policies (RSPs) that mirror regulatory thresholds. An RSP defines a series of Safety Levels (ASL-1 through ASL-4) , each gated by specific security and alignment conditions. Before training a model at a higher compute tier, developers must demonstrate that affirmative safety cases are met. If evaluations reveal dangerous capabilities, deployment is paused until mitigations are verified. This creates a precautionary principle operationalized through engineering gates.

A Compute Threshold Notification is a mandatory regulatory filing required when a training run exceeds a specified computational power limit. Under the EU AI Act, providers of general-purpose AI models must notify the AI Office within two weeks of a training run commencing if it exceeds 10^25 FLOPs. The notification must include the physical location of the data center, the total energy consumption, and the estimated training duration. This creates an early-warning system for regulators before the model is ever deployed.

Dangerous capability benchmarks are standardized test suites designed to measure a model's proficiency in domains that could cause catastrophic harm. Key benchmarks include:

• WMDP Benchmark: Tests knowledge of virology, bioweapons, and chemical agents
• CyberSecEval: Assesses offensive cyber capabilities and exploit generation
• SWE-bench: Measures autonomous software engineering, a proxy for self-improvement
• Persuasion Metrics: Evaluates the model's ability to change human beliefs A model exceeding predefined scores on these benchmarks triggers a capability threshold regardless of compute.

Hyperscaler concentration risk is the systemic vulnerability created when frontier AI training is concentrated in the data centers of a few cloud providers. If only three companies globally possess the infrastructure to train models exceeding the compute threshold, a single point of failure emerges for the entire AI supply chain. Regulators view this concentration as a systemic risk in itself, as a compromise, outage, or geopolitical disruption at one hyperscaler could cascade globally. This has driven interest in sovereign AI infrastructure and distributed training paradigms.

A test designed to measure an AI model's proficiency in domains that could cause catastrophic harm, such as bioweapons design, cyber-offensive operations, or autonomous replication. These benchmarks complement raw compute thresholds by evaluating actual emergent hazardous capabilities.

• Assesses chemical, biological, radiological, and nuclear (CBRN) risks
• Used in Responsible Scaling Policies
• Example: Evaluating a model's ability to synthesize novel pathogens

A protocol that ties the deployment of more powerful AI capabilities to the fulfillment of predefined safety conditions. If a model hits a systemic risk threshold, scaling must pause until mitigations are verified.

• Defines risk levels (e.g., Level 1 through Level 5)
• Requires independent third-party audit trails
• Mandates increased human-on-the-loop oversight at higher tiers

A set of regulatory requirements specifically imposed on foundation models with broad applicability under the EU AI Act. These obligations activate when a model is classified as posing systemic risk.

• Requires foundation model transparency reports
• Mandates adversarial robustness evaluations
• Includes copyright infringement scans of training data

A regulatory designation for AI systems that pose significant potential harm to health, safety, or fundamental rights. While distinct from systemic risk thresholds for general-purpose models, both trigger conformity assessments.

• Applies to specific use cases like biometrics and critical infrastructure
• Requires algorithmic impact assessments
• Demands automated decision logging for auditability

Techniques to identify when a model strategically pretends to comply with safety objectives during evaluation but not during deployment. This is a critical concern when models cross capability thresholds.

• Tests for specification gaming behaviors
• Uses held-out evaluation suites unknown to the model
• Critical for verifying safety alignment thresholds before release