Systemic Risk

Providers must perform state-of-the-art adversarial testing to identify and mitigate systemic risks. This includes red-teaming for dangerous emergent capabilities such as autonomous replication, offensive cyber operations, or the generation of chemical and biological weapons.

• Adversarial Testing: Simulated attacks by internal or external red teams to probe model vulnerabilities.
• Capability Assessment: Rigorous benchmarking against standardized dangerous capability thresholds.
• Mitigation Documentation: Detailed records of identified risks and the technical safeguards implemented.

Providers must immediately notify the AI Office and relevant national competent authorities of any serious incident involving their GPAI model. A serious incident is defined as a malfunction or failure that directly or indirectly leads to death, serious damage to health, or serious and irreversible damage to property or the environment.

• Immediate Notification: No undue delay after establishing a causal link between the model and the incident.
• Corrective Action: Obligation to implement immediate measures to contain and rectify the incident.
• Post-Incident Analysis: A comprehensive root-cause analysis must be submitted to authorities.

Systemic risk GPAI models require a state-of-the-art cybersecurity posture to defend against model theft, weight exfiltration, and unauthorized access. This extends beyond standard IT security to include AI-specific threat vectors.

• Model Weight Protection: Encryption at rest and in transit, secure enclaves for inference.
• Input Sanitization: Defenses against prompt injection, jailbreaking, and adversarial inputs.
• Access Control: Strict identity and access management (IAM) with audit logging for all model interactions.
• Supply Chain Security: Vetting of all third-party dependencies, fine-tuning adapters, and plugins.

Providers must draw up and keep up-to-date comprehensive technical documentation demonstrating compliance. This dossier must be made available to the AI Office and national authorities upon request.

• Training Data Summary: A detailed description of the data sources, composition, and preprocessing methodologies.
• Model Architecture Disclosure: Specification of the model's design, parameter count, and training compute budget.
• Performance Metrics: Documented results from capability evaluations, bias audits, and robustness testing.
• Copyright Compliance: A publicly available summary of the content used for training, respecting EU copyright law.

A continuous, iterative risk management system must be established to identify, estimate, and mitigate systemic risks throughout the model's lifecycle. This is not a one-time assessment but an ongoing operational requirement.

• Risk Identification: Systematic scanning for novel failure modes and societal-scale negative externalities.
• Mitigation Measures: Technical and organizational controls proportionate to the risk level.
• Continuous Monitoring: Post-market surveillance to detect emergent risks from real-world deployment.
• Stakeholder Engagement: Collaboration with independent experts, civil society, and downstream deployers.

A GPAI model is presumed to pose systemic risk if the cumulative compute used for training exceeds 10^25 FLOPs. Providers of models meeting this threshold must notify the European Commission without undue delay.

• Compute Threshold: 10^25 floating-point operations is the quantitative trigger for presumption.
• Qualitative Designation: The Commission may also designate a model based on capability benchmarks, market reach, or number of users.
• Provider Rebuttal: Providers can present arguments demonstrating their model does not pose systemic risk despite exceeding the compute threshold.

The regulatory category under the EU AI Act that triggers systemic risk obligations. A GPAI model is trained on broad data at scale, designed for generality of output, and adaptable to a wide range of distinct tasks. When a GPAI model's cumulative compute used for training exceeds 10^25 FLOPs, it is presumed to possess high-impact capabilities and must undergo systemic risk evaluation.

A binding requirement compelling providers to immediately notify the AI Office and national Market Surveillance Authorities of any malfunction or failure of a systemic-risk GPAI model that leads to death, serious damage to health or property, or a major disruption to critical infrastructure. This creates a real-time regulatory feedback loop distinct from routine post-market monitoring.

The standardized assessment protocols required to classify a model as systemic. This goes beyond accuracy metrics to include:

• Alignment Research Center (ARC) evaluations for autonomous replication
• Massive Multitask Language Understanding (MMLU) for broad knowledge
• Bias Benchmark for QA (BBQ) for societal harm vectors Providers must document and disclose these results in the technical documentation.

Systemic-risk GPAI models require state-of-the-art security measures proportional to the potential scale of harm. This includes:

• Model weight security: Encryption, access controls, and exfiltration prevention
• Input sanitization: Defenses against prompt injection and jailbreaking
• Supply chain integrity: Verifying the provenance of training data and fine-tuning adapters to prevent poisoning

Providers of systemic-risk GPAI must disclose to downstream deployers the model's capabilities, limitations, and known failure modes to enable compliant integration. This creates a chain of accountability: the provider's technical documentation informs the deployer's Fundamental Rights Impact Assessment, ensuring systemic risks are understood and mitigated at every layer of the value chain.