Child Safety Guardrail Violation Escalation Prompt

This prompt is for trust and safety engineers who need an automated first line of defense against child safety violations in user-generated content, AI-generated outputs, or uploaded media. It is designed to produce a structured, high-priority escalation record for any content that approaches or crosses CSAM-adjacent or child exploitation boundaries. The output is not a final decision. It is an evidence package that forces mandatory human review routing before any action is taken. Use this prompt when your platform has minor users, child-adjacent content, or regulatory exposure under frameworks such as NCMEC reporting requirements, the UK Online Safety Act, or the EU Digital Services Act.

Do not use this prompt for general content moderation, toxicity detection, or adult safety violations. Those require separate classification surfaces and escalation paths. This prompt assumes you have already defined your child safety policy taxonomy and have a human review queue ready to receive escalation records. The prompt is designed to be wired into a detection pipeline where upstream classifiers flag potential violations, and this prompt transforms those flags into adjudication-ready case files. It expects structured input including the flagged content, surrounding context, and the specific policy clause triggered.

Before deploying this prompt, ensure your human review team has clear decision guidelines for child safety cases, your legal team has reviewed the escalation language for regulatory compliance, and your logging infrastructure captures every escalation record immutably. Never use this prompt to auto-resolve or auto-remediate child safety cases. The only acceptable downstream action is routing to a qualified human reviewer with full case context. If your platform lacks a 24/7 on-call review capability for high-severity child safety escalations, do not deploy this prompt until that gap is closed.

This prompt is designed to operate as a system-level instruction for a model acting as a trust and safety classifier. Its job is to analyze a piece of content—a user message, a generated output, or a retrieved document—and determine if it crosses a defined child safety guardrail. When a violation is detected, the prompt must produce a structured escalation record that includes the violation type, severity, evidence excerpts, and explicit routing instructions for mandatory human review. The prompt uses square-bracket placeholders for all dynamic inputs, allowing you to wire it into a production moderation pipeline where the content, policy definitions, and risk thresholds are injected at runtime.

codeCopy
SYSTEM INSTRUCTION:
You are a Child Safety Guardrail Classifier for [PLATFORM_NAME]. Your sole function is to analyze the provided [CONTENT_TYPE] against the defined child safety policies and produce a structured output. You must never generate content that could be construed as harmful to minors. If a violation is detected, your output must trigger a mandatory human review. Do not make final adjudications.

POLICIES:
[CHILD_SAFETY_POLICIES]

INPUT CONTENT:
"""
[CONTENT]
"""

CONTEXT (Optional):
[CONTEXT]

OUTPUT_SCHEMA:
{
  "violation_detected": boolean,
  "violations": [
    {
      "policy_id": string,
      "violation_type": "CSAM_ADJACENT" | "CHILD_EXPLOITATION" | "GROOMING" | "AGE_RESTRICTED_CONTENT" | "OTHER",
      "severity": "CRITICAL" | "HIGH" | "MEDIUM",
      "confidence_score": number (0.0 to 1.0),
      "evidence_excerpt": string (exact quote from content, max 200 chars),
      "reasoning": string (brief explanation of why this violates the policy)
    }
  ],
  "escalation_required": boolean,
  "escalation_routing": {
    "queue_name": "child-safety-critical" | "child-safety-high" | "child-safety-review",
    "priority": "P0_IMMEDIATE" | "P1_URGENT" | "P2_STANDARD",
    "required_reviewer_role": "TRUST_AND_SAFETY_LEAD" | "COMPLIANCE_OFFICER" | "MODERATOR"
  },
  "content_should_be_quarantined": boolean,
  "summary_for_reviewer": string (one-sentence summary of the alert)
}

CONSTRAINTS:
- If no violation is detected, set "violation_detected" to false and return empty arrays for "violations".
- If confidence_score is below [CONFIDENCE_THRESHOLD], set "violation_detected" to false but flag for audit.
- Never include full CSAM-adjacent content in the evidence_excerpt. Truncate and note "[excerpt truncated for safety]".
- If the content involves a minor user, escalate regardless of confidence.
- All CRITICAL severity violations must set escalation_required to true and priority to P0_IMMEDIATE.

To adapt this template, replace each square-bracket placeholder with your production data. [CHILD_SAFETY_POLICIES] should contain your organization's specific policy definitions, including prohibited content categories and edge cases. [CONFIDENCE_THRESHOLD] is a numeric value (e.g., 0.7) that determines when ambiguous cases are escalated for audit rather than actioned. [CONTENT] is the raw text to be classified, and [CONTEXT] can include metadata such as user age, conversation history, or content source. The output schema is designed to be parsed by an application layer that routes the escalation to the correct review queue and quarantines content when content_should_be_quarantined is true. Before deploying, validate that the model reliably populates all required fields and that your downstream systems can handle the defined priority and queue_name values without manual mapping.

This prompt is not a standalone safety filter. It must be embedded inside a multi-layer detection pipeline where it serves as the final classification and evidence-packaging stage before human adjudication. The prompt expects pre-filtered content that has already passed through hash-matching, cryptographic perceptual hashing (e.g., PhotoDNA), and keyword-based triage. Its job is to take ambiguous or borderline content that survived earlier filters, classify it against your child safety policy taxonomy, assign a severity level, and produce a structured escalation record that a human reviewer can act on immediately. Do not send raw user uploads or unfiltered conversation streams directly to this prompt.

System architecture. Deploy this prompt behind a dedicated internal API endpoint that accepts only pre-screened content payloads. The calling service must attach a unique content_id, the detection source (e.g., chat_message, image_caption, user_profile), and any upstream filter scores. The prompt response must be parsed into a strict JSON schema before any downstream action. If parsing fails, retry once with a repair prompt. If parsing fails again, escalate the raw model output plus the content to a high-priority human queue with a PARSE_FAILURE flag. Never suppress a parse failure silently—every unparseable response in this domain is a potential missed detection. Log the full prompt, response, and parse result to an immutable audit store with a tamper-evident hash chain. Retain these logs for the duration required by your jurisdiction's child safety reporting regulations.

Validation and routing. After successful parsing, validate the output against a required field checklist: violation_detected (boolean), violation_category (from your approved taxonomy), severity (e.g., CRITICAL, HIGH, MEDIUM, LOW), evidence_excerpts (array of quoted content with byte offsets), policy_references (array of policy IDs), and confidence_score (0.0–1.0). If violation_detected is true and severity is CRITICAL or HIGH, route immediately to a dedicated child safety review queue with a 15-minute SLA. If violation_detected is true but severity is MEDIUM or LOW, route to the standard trust and safety queue. If violation_detected is false but confidence_score is below 0.85, route to a secondary review sampler that audits 10% of low-confidence negatives for false-negative drift. All other negatives are logged and closed. Never allow an automated action (deletion, suspension, report filing) based solely on this prompt's output without human confirmation.

Model choice and latency. Use a low-latency model with strong instruction-following and JSON output discipline. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are all suitable. Set temperature=0 to maximize output determinism. Set a strict timeout of 8 seconds; if the model does not respond within that window, fall back to a cached severity heuristic based on upstream filter scores and escalate the content with a MODEL_TIMEOUT flag. Do not retry timeouts more than once—a slow response in this pipeline is a safety risk. For high-throughput systems, pre-warm the model endpoint and use prompt caching on the static system instruction portion of the prompt to reduce median latency below 2 seconds.

What to avoid. Do not use this prompt as your only detection layer. Do not let the model decide whether to escalate—the harness must enforce escalation based on parsed severity. Do not log the full content payload in plaintext application logs; use reference IDs and store sensitive content in an encrypted evidence store with access controls. Do not skip the parse validation step, even if the model has been reliable in testing. And do not deploy this prompt without running the eval suite described in the Testing and Evaluation section against your specific policy taxonomy and a representative sample of borderline content from your platform.