Moderation API

Returns granular probability scores across distinct harm vectors rather than a single binary flag. Standard categories include hate speech, harassment, violence, self-harm, and sexual content.

• Granularity: Scores typically range from 0.0 to 1.0 per category.
• Thresholding: Clients set custom action thresholds (e.g., block if sexual/minors > 0.1).
• Sub-categories: Advanced APIs detect nuanced distinctions like 'identity attack' vs. 'threat'.

Operates as an isolated HTTP REST or gRPC endpoint that does not retain context between requests. This design ensures zero data leakage into conversational memory and allows for independent horizontal scaling.

• Latency Budget: Typically returns a verdict in < 100ms to avoid degrading the user experience.
• Decoupled Architecture: Runs independently of the generative model, allowing the main LLM to be swapped without rebuilding safety layers.

Evaluates both the user input (prompt) and the model output (response) through the same safety lens. This bidirectional check prevents the model from generating harmful content even if the user prompt appears benign.

• Input Guarding: Blocks adversarial prompts before they reach the core LLM.
• Output Guarding: Sanitizes or rewrites model responses that drift into unsafe territory.
• Round-Trip Policy: A single API key can be configured to enforce different strictness levels for inputs vs. outputs.

Specialized classifiers trained to detect expressions of suicidal ideation, self-harm intent, or eating disorder triggers with extremely high recall. These categories bypass standard block logic to trigger a compassionate, resource-providing response rather than a generic error.

• Crisis Hotline Injection: Automatically appends localized mental health resources to the response.
• Escalation Flags: Generates internal alerts for human review if persistent patterns are detected.

Detects policy violations across 100+ languages and mixed-language inputs (code-switching). Modern moderation APIs use multilingual embedding models to catch toxicity that translation-based systems miss.

• Script Agnostic: Works on Latin, Cyrillic, Arabic, and CJK scripts.
• Adversarial Spelling: Detects leetspeak, intentional misspellings, and homoglyph attacks (e.g., 'h@te').

Evaluates real-time streaming output in sliding windows or sentence chunks rather than waiting for the full response. This prevents toxic content from being displayed to the user during token-by-token generation.

• Early Termination: Issues an abort signal to the LLM mid-stream if a violation is detected.
• Buffer Management: Balances chunk size against latency to maintain fluid UX while ensuring safety coverage.