Safety Classifier

The primary function of a safety classifier is to detect and categorize specific types of harmful content. It is typically a multi-label classifier trained on datasets annotated for various harm categories.

• Common Categories: Toxicity, hate speech, violence, self-harm, sexual content, unethical advice, privacy violations, and misinformation.
• Output: For a given text input, the classifier outputs a probability score for each harm category (e.g., violence: 0.92, hate_speech: 0.15).
• Example: The classifier would flag the prompt "Write a plan to hack into a corporate network" with high scores for unethical_advice and potentially violence.

Safety classifiers are often encoder-only models (like BERT) or smaller, distilled versions of large language models (LLMs), fine-tuned separately from the main generative model.

• Architecture: They are designed for sequence classification, taking text as input and outputting logits for predefined classes.
• Training Data: Requires large, carefully curated datasets of harmful and benign examples, often involving synthetic data generation and automated red-teaming to cover edge cases.
• Fine-Tuning: The base model is fine-tuned using standard supervised learning on these labeled datasets to maximize accuracy on the harm detection task.

A safety classifier operates as a modular component within a larger AI system, typically invoked during the input validation or output verification stages of the inference pipeline.

• Pre-Generation Check: The user's prompt is analyzed before being sent to the main LLM. If a harm threshold is exceeded, a refusal mechanism can be triggered.
• Post-Generation Check: The LLM's proposed output is scanned before being shown to the user. Non-compliant outputs can be blocked, filtered, or sent back for revision via a self-critique loop.
• Governance Hook: The classifier acts as a governance hook, a piece of middleware that enforces policy-as-code by intercepting and evaluating all traffic.

The raw probability scores from the classifier are compared against configurable thresholds to make binary allow/block decisions, translating detection into actionable policy.

• Configurable Sensitivity: Different harm categories can have different thresholds (e.g., a lower tolerance for violence than for mild profanity).
• Policy-as-Code: These thresholds and the logic for combining multiple category scores are defined as executable code, enabling precise, auditable enforcement.
• Explainable Refusal: When a request is blocked, the system can reference the specific category and score that triggered the refusal (e.g., "This request was blocked due to a high probability of containing violent content.").

Effective safety classifiers must be rigorously evaluated and hardened against evasion techniques, a core concern for adversarial robustness.

• Evaluation Metrics: Performance is measured using precision, recall, and F1-score on held-out test sets, with a focus on minimizing false negatives (missed harms).
• Jailbreak Detection: The classifier must be robust against adversarial prompts and jailbreak attempts designed to bypass its filters through obfuscation or social engineering.
• Continuous Testing: Requires ongoing evaluation with new automated red-teaming techniques to discover and patch vulnerabilities in the classification logic.

In a Constitutional AI framework, the safety classifier is a key tool for operationalizing the principles of the constitution. It provides the measurable, automated check required for scalable alignment.

• Principle Adherence Scoring: The classifier's scores serve as a quantitative measure of principle adherence for a given text.
• Enabling RLAIF: In Reinforcement Learning from AI Feedback (RLAIF), a safety classifier can act as the source of AI-generated preferences, rewarding outputs it scores as safe and penalizing harmful ones.
• Audit Trail Generation: The classifier's decisions and scores are logged as part of an audit trail, providing a verifiable record of safety checks performed on each interaction.