Invented Customer Data Removal Prompt for Support Tickets

This prompt is a repair tool, not a generation tool. It belongs in the workflow after a model has drafted a customer support response from CRM context but before that draft is reviewed by a human agent or delivered to the customer. Its job is to catch the most dangerous class of hallucination in support systems: fabricated customer data. When a model invents a customer name, order number, subscription tier, or prior interaction detail that does not exist in the source CRM record, the result is not just a bad answer—it is a trust-destroying error that can trigger privacy incidents, compliance violations, and escalations. This prompt takes the draft response and the CRM record that should have grounded it, identifies every piece of invented data, strips it, replaces it with explicit placeholders, and logs each removal with a reason code so your team can monitor hallucination patterns over time.

Use this prompt when your support AI generates drafts from retrieved CRM context and you have observed hallucinations in customer identifiers, account attributes, or interaction history. It is designed for systems where the CRM record is the single source of truth and any customer-facing data not present in that record is considered fabricated. The prompt expects two inputs: the model-generated draft response and the structured CRM record containing verified customer fields. It returns a cleaned response with invented data replaced by placeholders like [VERIFY CUSTOMER NAME] and a structured removal log that your application can store for quality monitoring. This is not a prompt for general fact-checking, tone adjustment, or response quality scoring—it is narrowly scoped to invented customer data detection and removal.

Do not use this prompt as your primary defense against hallucination. It is a safety net, not a prevention mechanism. If your model frequently invents customer data, the root cause is likely in your retrieval pipeline, prompt construction, or context grounding—fix those first. This prompt also cannot detect subtle errors where the model uses a real customer name but associates it with the wrong account or interaction. It only flags data that has no match in the provided CRM record. For high-risk support domains such as healthcare, financial services, or insurance, always route cleaned outputs through human agent review before customer delivery. The removal log this prompt produces should feed into your observability stack so you can track hallucination frequency by model version, prompt version, and support category.

This prompt template is designed to be placed as a post-generation repair step in your support ticket workflow. After your primary model drafts a response, this prompt acts as a safety net, comparing the draft against a verified CRM record. Its job is to identify any customer-specific information the model fabricated—such as a name, email, order number, or past interaction—and remove it, replacing it with a [REDACTED] token and logging the removal. The prompt is structured to produce a clean, safe response and a structured audit log in a single JSON output, making it easy to integrate into an automated pipeline.

textCopy
You are a data integrity auditor for a customer support system. Your task is to compare a drafted support response against a verified CRM record and remove any customer-specific information that was invented by the model rather than retrieved from the record.

# INPUTS
- [DRAFT_RESPONSE]: The model-generated support ticket reply.
- [CRM_RECORD]: The verified customer data retrieved from the system of record. This is the only source of truth.

# DEFINITION OF INVENTED DATA
Invented data is any customer-specific information in the draft that is not explicitly present in the CRM record. This includes, but is not limited to:
- Personally Identifiable Information (PII): Names, emails, phone numbers, physical addresses.
- Account Details: Account IDs, subscription tiers, billing amounts, payment methods.
- Interaction History: Dates of past contacts, names of previous agents, details of prior cases.
- Order Information: Order numbers, product names, shipping details, tracking numbers.

# INSTRUCTIONS
1.  Analyze the [DRAFT_RESPONSE] sentence by sentence.
2.  For every piece of customer-specific information, check if it is present in the [CRM_RECORD].
3.  If the information is NOT in the [CRM_RECORD], it is invented. Replace the invented text with the token `[REDACTED]`.
4.  Do not modify any text that is generic advice, a greeting, a closing, or information that is correctly sourced from the [CRM_RECORD].
5.  Do not redact information that is correctly present in the [CRM_RECORD].

# OUTPUT_SCHEMA
Return a single valid JSON object with the following keys:
- `clean_response`: The full text of the draft response with all invented customer data replaced by `[REDACTED]`.
- `audit_log`: An array of objects, each representing a redaction. Each object must have the keys:
  - `original_text`: The exact text that was removed.
  - `reason`: A brief explanation of why it was flagged (e.g., "Email not in CRM record").
  - `location`: A string describing where in the text the redaction occurred (e.g., "Second paragraph, first sentence").

# CONSTRAINTS
- Base all decisions strictly on the [CRM_RECORD]. If you are unsure, flag it for redaction.
- Do not add any new information to the `clean_response`.
- Do not alter the structure or tone of the original draft beyond the redactions.

To adapt this template, replace the [DRAFT_RESPONSE] and [CRM_RECORD] placeholders with your actual data. The [CRM_RECORD] should be a structured text block containing all verified fields you have on the customer. For high-risk deployments, the audit_log is critical; pipe it to your observability stack to monitor the frequency and type of redactions. If the audit_log is not empty, consider routing the final clean_response for a quick human review before sending, as a high redaction rate can indicate a deeper prompt engineering or retrieval issue upstream.

This prompt is designed to operate as a post-generation filter within a customer support ticketing pipeline. It should be invoked after a draft reply is generated by a model—whether from a RAG system, a fine-tuned assistant, or a general-purpose LLM—and before that reply is sent to a customer or written to the CRM. The harness must provide the draft reply as [DRAFT_REPLY] and the ground-truth customer context as [CRM_CONTEXT]. The [CRM_CONTEXT] should be a structured JSON object containing the verified fields your system of record holds: customer_name, account_number, email, phone, recent_tickets, subscription_tier, and any other fields the model is allowed to reference. The prompt's job is to strip anything that appears in the draft but not in this context, not to enrich the reply with missing data.

The implementation should follow a strict validate-then-log pattern. After receiving the prompt's JSON output, parse the cleaned_reply and removals array. Run a structural validator that confirms: (1) every entry in removals has a non-empty field, invented_value, and reason; (2) no invented_value from the removals list appears in the cleaned_reply text; (3) the cleaned_reply contains no PII patterns (emails, phone numbers, account IDs) that are absent from [CRM_CONTEXT]. If validation fails, do not retry with the same prompt—escalate to a human reviewer with the original draft, the CRM context, and the failed validation report. If validation passes, log the full removals array to your quality monitoring system (e.g., a hallucination_log table or Datadog custom metric) keyed by ticket_id and agent_id. This log becomes your primary signal for model drift, RAG retrieval gaps, and prompt effectiveness over time.

For high-risk deployments—such as financial services, healthcare, or any workflow where a fabricated customer detail could cause regulatory exposure—add a human approval gate for any reply where removals is non-empty. Route the cleaned reply and the removals diff to a review queue. The reviewer should see a side-by-side view: original draft on the left, cleaned reply on the right, with invented fields highlighted. Only after explicit approval should the reply enter the send path. For lower-risk deployments, you can auto-send replies with zero removals and sample a percentage of replies with removals for human audit. Do not silently drop removals without logging; that destroys the observability you need to improve the upstream generation prompt or retrieval step.

Model choice matters here. This prompt works best with models that have strong instruction-following and JSON mode support, such as GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro. Avoid using smaller or older models that may hallucinate the removals list itself. Set temperature=0 and enable structured output mode (JSON schema constraint) if your provider supports it. The output schema should be enforced at the API level, not just described in the prompt. If your model provider does not support strict JSON mode, add a retry layer: if the response fails to parse as valid JSON matching the expected schema, retry once with the same inputs and a stronger instruction prefix. If the second attempt fails, escalate to human review rather than risking a malformed filter passing bad data through.

Finally, treat this prompt as one component in a defense-in-depth strategy. It catches fabrication after generation, but it does not prevent the model from inventing data in the first place. Pair it with upstream improvements: tighten your RAG retrieval to ensure CRM context is complete and correctly chunked, add a system instruction that prohibits generating customer details not present in the provided context, and monitor the removals log for patterns. If you see the same field being invented repeatedly (e.g., the model keeps fabricating a last_contact_date), that's a signal to improve your context assembly or add an explicit negative example to the generation prompt. The removal prompt is your safety net, not your primary defense.