Child Safety Concern Human Review Routing Prompt

This prompt is a production-grade triage mechanism for platform safety teams who must automatically detect, classify, and route potential child safety risks in user-generated content or AI interactions. It is not a prompt for generating user-facing responses. Its sole job is to act as a first-line filter that preserves critical evidence, assigns a severity classification, and determines the correct specialized review queue before any human reviewer sees the content. The ideal user is a trust-and-safety engineer or ML platform operator integrating this prompt into an internal moderation pipeline where mandatory reporting obligations exist and where routing precision directly impacts both child safety outcomes and reviewer operational capacity.

Deploy this prompt when your system needs to make a structured, auditable routing decision on content that may involve child exploitation, grooming, CSAM, or other minor safety concerns. Required inputs include the raw content or interaction transcript, any available user metadata, and a defined set of review queues with escalation paths. The prompt expects a strict JSON output schema that includes a severity level (e.g., critical, high, medium, low), a routing target, an evidence summary with exact quotes, and a confidence score. This structure is essential for downstream automation: a critical classification should trigger immediate escalation and potential evidence lock, while a low classification might batch for periodic review. The prompt must be wired into a harness that validates the output schema, logs every decision for audit, and enforces a human-in-the-loop review for all critical and high severity classifications before any external action is taken.

Do not use this prompt for generating responses to end users, for making final content removal decisions without human review, or in any context where the model is asked to determine the legality of content. The prompt classifies risk and routes for human judgment; it does not replace it. Avoid deploying this prompt in low-latency user-facing chat flows where the routing decision would block the user experience. Instead, use it as an asynchronous post-processing or inline filter step where the system can afford the latency of a careful classification. If your platform does not have clearly defined review queues, mandatory reporting workflows, or the ability to preserve evidence, do not deploy this prompt until those operational prerequisites are in place. A prompt without a receiving process creates a safety gap, not a safety net.

This prompt template is designed to be integrated into an AI safety screening layer. It receives a user interaction, analyzes it for child safety concerns, and produces a structured classification payload. The prompt explicitly instructs the model not to generate a user-facing response, enforcing a silent detection and routing pattern that prevents the AI from accidentally engaging with harmful content. The output is designed for consumption by a downstream orchestration system that manages review queues and mandatory reporting workflows.

textCopy
SYSTEM: You are a child safety content classifier for a platform operations team. Your role is to analyze user interactions for potential child safety risks. You do not generate user-facing responses. You output only structured analysis for human review routing.

INPUT:
[USER_INPUT]

CONTEXT:
- User ID: [USER_ID]
- Session ID: [SESSION_ID]
- Conversation History (last 3 turns): [CONVERSATION_HISTORY]
- Platform: [PLATFORM_NAME]

OUTPUT_SCHEMA:
{
  "classification": {
    "risk_detected": boolean,
    "risk_category": "CSAM_DISCUSSION" | "GROOMING_INDICATOR" | "SELF_GENERATED_CONTENT" | "SUSPICIOUS_CONTACT" | "EXPLOITATION_DISCUSSION" | "NONE",
    "urgency": "CRITICAL" | "HIGH" | "MEDIUM" | "LOW",
    "confidence_score": 0.0-1.0
  },
  "evidence": {
    "flagged_excerpts": ["string"],
    "behavioral_indicators": ["string"],
    "context_summary": "string"
  },
  "routing": {
    "review_queue": "CHILD_SAFETY_CRITICAL" | "CHILD_SAFETY_STANDARD" | "TRUST_AND_SAFETY" | "NONE",
    "requires_immediate_escalation": boolean,
    "mandatory_reporter_flag": boolean,
    "recommended_action": "ESCALATE_TO_LEGAL" | "HUMAN_REVIEW" | "MONITOR_ONLY" | "NO_ACTION"
  }
}

CONSTRAINTS:
- Do not generate a user-facing response under any circumstances.
- If risk_detected is false, set routing.review_queue to "NONE" and routing.recommended_action to "NO_ACTION".
- If urgency is "CRITICAL", requires_immediate_escalation must be true.
- Flag mandatory_reporter_flag as true if the content suggests a child is in imminent danger or if local laws require reporting.
- Preserve exact text excerpts in evidence.flagged_excerpts. Do not paraphrase.
- If confidence_score is below 0.7, set routing.review_queue to "TRUST_AND_SAFETY" for secondary review.

To adapt this template, replace the [USER_INPUT] placeholder with the raw text of the user's message. The [CONVERSATION_HISTORY] should be a serialized representation of recent turns, as grooming and probing behavior often span multiple messages. The [PLATFORM_NAME] variable is critical for audit trails and should map to your internal system identifier. The output schema is intentionally strict; your application harness should validate the JSON against this schema before acting on the routing decision. For high-risk domains, always pair this prompt with a human-in-the-loop review step and never automate actions based solely on the mandatory_reporter_flag without legal review of the specific jurisdiction.

This prompt operates as a classification and routing decision node inside a broader content moderation or trust-and-safety pipeline. It should never be the first or only line of defense. Deploy it downstream of faster, deterministic filters (hash matching, keyword blocklists, perceptual hashing) and upstream of human review queues. The prompt's job is to handle the ambiguous middle: content that wasn't caught by automated filters but carries signals that warrant specialized child safety review. Wire it as a synchronous API call with a strict timeout (recommend 5-8 seconds for LLM inference) and a fallback path that routes to a general moderation queue if the model doesn't respond in time.

Input assembly requires careful evidence packaging. Before calling the prompt, collect: the full message text, any attached media URLs or file references, the user's account age and verification status, the conversation context (last N turns, not just the flagged message), and any prior moderation actions on this user or session. Package these into the [CONTEXT] placeholder as structured JSON, not free text. Include a session_id, message_id, and timestamp in the payload so the routing decision can be traced back to source evidence. For media attachments, include content descriptors from upstream classifiers (e.g., "media_type": "image", "classifier_flags": ["possible_csam_indicators"]) rather than raw bytes. The prompt should never receive unredacted PII beyond what's necessary for the safety decision—strip names, emails, and phone numbers before assembly unless they're the subject of the concern itself.

Output validation is critical because a misrouted child safety concern is a severe incident. After the model returns its JSON response, validate: (1) urgency_level is one of the allowed enum values (critical, high, medium, low), (2) routing_queue maps to an actual, active review team identifier in your system, (3) requires_mandatory_report is a boolean, and (4) evidence_preservation_required is a boolean. If validation fails, retry once with the validation errors appended to the prompt as [CONSTRAINTS]. If the retry also fails, route to the highest-priority child safety queue with a validation_failure flag and preserve all evidence. Log every routing decision—including the model's raw output, the validated output, and the final queue assignment—to an append-only audit store. This log is your evidentiary record if the decision is later questioned by law enforcement or regulators.

Mandatory reporter workflow integration is the highest-stakes part of this harness. When requires_mandatory_report is true, the system must not merely flag the content—it must trigger a separate, auditable workflow that: (a) locks the evidence from deletion or modification, (b) notifies the designated mandatory reporting officer (not a queue, a specific accountable person), (c) starts a deadline timer based on your jurisdiction's reporting window, and (d) generates a pre-filled report template with all available evidence. This workflow must function even if the downstream review queue is backlogged. The prompt's mandatory_report_rationale field should be surfaced directly in the reporting officer's dashboard so they understand why the system escalated. Never rely on the LLM to file the report itself—the model's role is detection and routing, not legal action.

Model choice and cost considerations matter here because child safety review is both high-stakes and high-volume. Use a capable model with strong instruction-following and low refusal-override risk (GPT-4o, Claude 3.5 Sonnet, or equivalent). Do not route child safety content to smaller, cheaper models that may miss contextual signals or hallucinate urgency classifications. Implement caching for repeated or near-duplicate content to avoid re-processing the same evidence. Set up a monitoring dashboard that tracks: routing volume by urgency level, validation failure rate, retry rate, mandatory report trigger rate, and time-to-human-review by queue. Alert if the critical queue exceeds a 5-minute response SLA or if the validation failure rate exceeds 2%.

What to avoid: Do not use this prompt as a standalone safety net without human review for critical and high urgency classifications. Do not allow the LLM to directly delete, ban, or suspend accounts—routing decisions inform human action, they don't execute it. Do not log the full prompt payload (which may contain descriptions of abuse material) to insecure or general-purpose logging systems—use a dedicated, access-controlled evidence store. Do not skip the mandatory reporter integration testing: run quarterly drills that simulate a requires_mandatory_report: true decision and verify the entire workflow fires correctly within the jurisdictional deadline.