Risk Management System

Risk management is not a one-time pre-market activity. It is a continuous, iterative process planned and executed throughout the entire lifecycle of a high-risk AI system.

• Pre-Market: Risk identification and estimation during design and development.
• Post-Market: Systematic monitoring and review of real-world data to identify emerging risks.
• Feedback Loop: Post-market findings must feed back into the risk assessment to trigger updates, retraining, or substantial modification assessments.

The EU AI Act mandates a structured four-step methodology for risk assessment:

1. Identification: Systematically identify the reasonably foreseeable risks to health, safety, and fundamental rights.
2. Estimation: Analyze and evaluate the severity and probability of each identified risk occurring.
3. Evaluation: Judge whether the residual risk is acceptable by comparing it against the intended purpose and the state of the art.
4. Mitigation: Adopt appropriate and targeted risk management measures to eliminate or reduce risks to an acceptable level.

The ultimate goal is to reduce risk to an acceptable residual level. A risk is considered acceptable when:

• All reasonably foreseeable risks have been identified and estimated.
• All relevant harmonized standards and state-of-the-art mitigation techniques have been applied.
• The remaining risk is judged to be justified by the system's benefits and is communicated transparently to the deployer.
• The system's design inherently minimizes risk as much as possible, with mitigation measures layered on top.

Risk mitigation measures must be validated through rigorous testing. This includes testing against reasonably foreseeable misuses and system failures.

• Adversarial Testing: Simulating malicious inputs, evasion attacks, and data poisoning attempts.
• Edge Case Analysis: Evaluating performance on outlier data points and rare but high-impact scenarios.
• Fail-Safe Design: Verifying that the system enters a safe state or transfers control to a human operator upon detecting a critical anomaly or distributional shift.

The risk management system is a core component of the mandatory Quality Management System (QMS) for providers. It cannot exist in isolation.

• Documented Procedures: Risk management policies, techniques, and protocols must be fully documented and version-controlled.
• Traceability: Every identified risk must be traceable to a specific mitigation measure and a corresponding verification test.
• Executive Accountability: The QMS must define clear management responsibility for signing off on residual risk acceptance and authorizing market placement.

The risk management output directly informs the instructions for use. Providers must transparently communicate residual risks to the deployer.

• Known Limitations: Clearly state the specific conditions or populations for which the system's performance is unverified or degraded.
• Required Oversight: Specify the necessary human oversight measures the deployer must implement to manage residual risk.
• Incident Reporting: Provide clear procedures for the deployer to report serious incidents or malfunctions back to the provider for post-market monitoring.