Misinformation and Disinformation Risk Triage Prompt

This prompt is a classification gateway for platform trust and safety teams who need to triage user-generated content, posts, or messages for misinformation and disinformation risks before the content is amplified, recommended, or allowed to remain visible. It scores content for verifiably false claims, coordinated inauthentic behavior (CIB) signals, and manipulation context. Use it during high-risk events such as elections, public health crises, or breaking news cycles where the cost of delayed detection is high. The prompt is designed to sit inside a classification gateway: it reads the content and any available context, then outputs a structured risk assessment that downstream systems can use to route the content for fact-checking, quarantine, human review, or automated enforcement.

Do not use this prompt as a standalone fact-checker. It does not verify claims against a live knowledge base. It assesses the risk profile of the content and recommends the next processing step. Always pair it with a fact-checking pipeline or human review queue for high-severity outputs. The ideal user is a trust and safety engineer or a moderation operations lead who needs a consistent, auditable signal to scale their team's triage capacity. The prompt requires the raw content text and, optionally, user history or thread context to assess coordination patterns. Without this context, CIB detection accuracy will degrade.

Before wiring this prompt into a production system, define the downstream routing map. A high_risk score should trigger quarantine and human review, not an automated takedown. A medium_risk score might route to a fact-checking queue. A low_risk score can allow the content to proceed with standard moderation. The prompt's value is in reducing the volume of content that requires expensive human review while ensuring high-risk content is never auto-approved. Never use the model's raw output to take irreversible enforcement actions without a human-in-the-loop step for high-severity classifications.

The following prompt template is designed to be pasted directly into your classification gateway. It instructs the model to act as a trust and safety triage agent, analyzing a piece of content for verifiably false claims, signals of coordinated inauthentic behavior (CIB), and manipulation context. The output is a structured JSON object that your application can parse to make a deterministic routing decision—escalating to a human review queue, sending to a fact-checking partner API, or quarantining the content entirely.

codeCopy
You are a trust and safety triage agent. Analyze the provided content for misinformation, disinformation, and manipulation risks. Your analysis must be grounded strictly in the content and the provided context. Do not infer intent beyond what is stated.

INPUT:
[CONTENT]

CONTEXT:
- Current Event: [EVENT_NAME]
- Known False Narratives: [KNOWN_NARRATIVES]
- Author/Account Risk Score: [ACCOUNT_RISK_SCORE]

OUTPUT_SCHEMA:
{
  "risk_tier": "low" | "medium" | "high" | "critical",
  "verifiably_false_claims": [
    {
      "claim": "string",
      "fact_check_status": "false" | "misleading" | "unsubstantiated" | "needs_review",
      "evidence_reference": "string or null"
    }
  ],
  "cib_signals": [
    {
      "signal": "string",
      "indicator_type": "coordinated_phrasing" | "brigading_pattern" | "inauthentic_amplification" | "network_synchrony",
      "confidence": 0.0-1.0
    }
  ],
  "manipulation_context": {
    "target_audience": "string or null",
    "emotional_valence": "fear" | "anger" | "outrage" | "hope" | "neutral",
    "virality_risk": 0.0-1.0
  },
  "routing_decision": "clear" | "fact_check" | "human_review" | "quarantine",
  "routing_reasoning": "string"
}

CONSTRAINTS:
- If no verifiably false claims are detected, the `verifiably_false_claims` array must be empty.
- `virality_risk` must be scored based on emotional manipulation and current event context, not just engagement metrics.
- A `routing_decision` of "quarantine" requires a `risk_tier` of "critical" and at least one high-confidence CIB signal.
- Do not fabricate evidence references. Use `null` if no specific source can be cited.

To adapt this template for your environment, replace the square-bracket placeholders with real data before each inference call. The [CONTENT] placeholder should receive the full text of the post, message, or article. The [EVENT_NAME] and [KNOWN_NARRATIVES] fields should be populated from an external knowledge base or feature flag that your operations team updates during high-risk events, such as an election or public health crisis. The [ACCOUNT_RISK_SCORE] should be a pre-computed value from your internal identity and behavioral analysis system, injected here to provide the model with additional signal without requiring it to perform account-level analysis on unstructured data.

Before deploying this prompt, validate that your application layer can parse the JSON output strictly against the defined schema. Any deviation—such as a missing routing_decision or an invalid risk_tier—should trigger a retry or a fallback to a human review queue. In high-stakes scenarios, log every output alongside the input context for auditability, and implement an eval pipeline that measures precision and recall against a golden dataset of known misinformation examples to catch drift in the model's classification behavior.

This prompt is designed as a synchronous classification step within a broader content evaluation pipeline. It should be invoked after content ingestion but before any automated action (such as labeling, demotion, or user notification) is taken. The prompt expects a single piece of content and its surrounding context as input, and it returns a structured JSON risk assessment. In production, this prompt is not a standalone decision-maker; it is a signal generator whose output must be validated, logged, and combined with other signals (user reputation, virality metrics, prior strikes) before a final moderation decision is made.

To integrate this prompt, wrap it in a service that enforces a strict JSON output contract. If the model returns malformed JSON, missing required fields, or values outside the defined enums, the service should retry the request once with a stronger formatting instruction appended to the original prompt. If the retry also fails, the content should be routed to a human review queue with a VALIDATION_FAILURE flag. Log every request and response, including the raw prompt, model version, latency, and confidence scores, to an immutable audit store. This is critical for trust and safety workflows where decisions must be explainable to regulators, users, and internal policy teams. For model choice, use a capable frontier model (such as GPT-4o or Claude 3.5 Sonnet) with low temperature (0.1–0.2) to maximize deterministic scoring. Avoid using small or fine-tuned models for this task unless you have a rigorous evaluation pipeline that proves they match frontier model performance on your specific misinformation taxonomy.

The output schema includes a requires_human_review boolean. Your harness must respect this flag unconditionally: if it is true, do not proceed with any automated action, regardless of other scores. Additionally, implement a hard rule that any content scoring above a virality_risk threshold of 8 or a manipulation_score above 7 is automatically escalated. For high-stakes events (elections, public health crises), consider routing all content with a disinformation_risk score above 5 for human review, overriding the model's own recommendation. The harness should also check for the presence of fact_check_urls in the output; if the model provides them, your system should verify that the URLs are from an approved, vetted list of fact-checking organizations before surfacing them to human reviewers or end-users. Never display unverified external URLs directly from the model output.

Before deploying any change to this prompt or its harness, run a regression suite against a golden dataset of 500+ labeled examples covering confirmed misinformation, legitimate news, satire, opinion, and edge cases like misleading imagery captions. Measure precision and recall for each risk tier, and require a human review of 50 random samples from each scoring bucket to calibrate thresholds. In production, monitor the distribution of risk scores daily; a sudden shift in the ratio of high-risk to low-risk classifications may indicate prompt drift, a change in the input content mix, or a model behavior change. Set up alerts for when the requires_human_review rate exceeds 20% of traffic, as this may signal a model failure or a coordinated attack. Finally, ensure your harness can be disabled via a kill switch without affecting the rest of the content pipeline, allowing for emergency rollback during incidents.