Multi-Turn Slot Filling Trace Inspection Prompt

This prompt is designed for developers and QA engineers responsible for conversational AI systems that collect structured data from users across multiple turns, such as booking assistants, insurance intake forms, or customer onboarding flows. The job-to-be-done is a rigorous, turn-by-turn audit of slot-filling behavior: you need to verify that every piece of information the user provided was correctly captured, that no slot was silently overwritten or dropped, and that the final collected state faithfully represents the user's stated requirements. This is not a prompt for designing the slot-filling logic itself; it is a diagnostic tool for inspecting a completed production trace to find silent data loss and state corruption.

Use this prompt when you have a full, exported trace of a multi-turn conversation that includes user utterances, assistant responses, and the internal slot state at each turn boundary. The prompt requires a structured input containing the turn history, the slot schema definition, and the final collected state object. It is most effective when applied to traces that have already been flagged by user complaints, QA failures, or anomaly detection systems. The output is a detailed slot-state audit report that identifies specific turns where slots were missed, overwritten, or contradicted, and compares the final state against the user's explicit requirements. This is a high-signal diagnostic tool, not a real-time correction mechanism.

Do not use this prompt for real-time slot-filling logic, for conversations with fewer than three turns, or for systems where slot state is not logged at each turn boundary. It is also inappropriate for diagnosing retrieval quality, latency issues, or persona drift—those require separate specialized prompts. The prompt assumes that the trace data is complete and accurate; if your logging is lossy or the slot schema is undocumented, the audit will produce unreliable results. For high-risk domains such as healthcare intake or financial applications, always pair the automated audit with a human review step before taking corrective action on the identified discrepancies.

This prompt template is designed to be dropped into your trace analysis pipeline whenever you need to audit a multi-turn slot-filling session. It accepts a full turn-by-turn trace export and a target schema of expected slots, then produces a structured audit report comparing the final collected state against the user's stated requirements. The prompt is model-agnostic but assumes the trace includes user utterances, assistant responses, and any internal slot-state snapshots your system logs.

textCopy
You are an AI trace auditor specializing in multi-turn slot-filling conversations. Your task is to inspect a complete conversation trace and produce a slot-state audit report.

## INPUT
- Conversation trace: [TRACE_JSON]
- Target slot schema: [SLOT_SCHEMA]
- User's stated requirements (extracted from final turn or summary): [USER_REQUIREMENTS]

## TASK
For each slot defined in the target schema, trace its value across every turn in the conversation. Identify:
1. The turn where the slot was first collected.
2. Any turn where the slot value was overwritten or modified.
3. Any turn where the slot value was contradicted by the user but not updated.
4. The final value of the slot at the end of the session.
5. Whether the final value matches the user's stated requirements.

## OUTPUT SCHEMA
Return a JSON object with this structure:
{
  "session_id": string,
  "audit_timestamp": string,
  "slots": [
    {
      "slot_name": string,
      "expected_in_schema": boolean,
      "first_collected_turn": number | null,
      "value_timeline": [
        {
          "turn": number,
          "value": string | null,
          "source": "user" | "assistant_inferred" | "system_default" | "tool_output",
          "event": "collected" | "overwritten" | "contradicted_not_updated" | "confirmed"
        }
      ],
      "final_value": string | null,
      "matches_user_requirements": boolean | "unclear",
      "issues": [string]
    }
  ],
  "summary": {
    "total_slots_expected": number,
    "slots_collected": number,
    "slots_missing": [string],
    "slots_with_contradictions": [string],
    "slots_overwritten": [string],
    "overall_assessment": "clean" | "minor_issues" | "major_data_loss"
  }
}

## CONSTRAINTS
- Do not invent slot values not present in the trace.
- If a slot was never mentioned, mark first_collected_turn as null and final_value as null.
- Flag any slot where the assistant's final value differs from what the user explicitly stated.
- If the trace contains internal state snapshots, prefer those over inferring values from assistant utterances.
- Mark matches_user_requirements as "unclear" if the user's requirement is ambiguous or unstated.

## EXAMPLES
[EXAMPLES]

## RISK_LEVEL
[RISK_LEVEL]

To adapt this template for your system, replace the square-bracket placeholders with your actual data. [TRACE_JSON] should contain the full turn-by-turn log, ideally with user utterances, assistant responses, and any internal slot-state snapshots your application logs. [SLOT_SCHEMA] defines the expected slots—this can be a JSON Schema fragment or a simple list of slot names with types. [USER_REQUIREMENTS] should be extracted from the user's final confirmation or a summary of their stated needs across the session. The [EXAMPLES] placeholder should be populated with one or two annotated traces showing correct and incorrect slot-filling behavior to calibrate the model's judgment. Set [RISK_LEVEL] to "high" if the slot values drive financial transactions, bookings, or compliance decisions—this triggers stricter contradiction detection and flags unclear matches for human review.

Before deploying this prompt in production, validate its output against a golden set of manually audited traces. Common failure modes include the model hallucinating slot values from assistant responses when no internal state snapshot is available, missing contradictions that span non-adjacent turns, and over-flagging minor phrasing differences as contradictions. Run this prompt on at least 20 labeled traces and compare the audit results against human annotations before relying on it for automated quality gates. For high-risk domains, always route traces flagged with "major_data_loss" or "slots_with_contradictions" to a human reviewer.

This prompt is designed to operate as a batch trace analysis job, not a real-time synchronous call. You feed it a complete multi-turn trace export and receive a structured slot-state audit. The implementation harness must handle trace extraction, prompt assembly, model invocation, output validation, and routing of results to the appropriate downstream system—typically an observability dashboard, an alerting pipeline, or a manual review queue. Because the prompt inspects conversational data that may contain user PII, the harness must enforce data minimization: redact or pseudonymize user utterances before they reach the model unless the analysis specifically requires verbatim text for slot-value extraction.

Wiring the prompt into an application starts with a trace export adapter. Your observability platform or conversation store must provide a standardized trace format containing: the full turn history with user utterances and assistant responses, the slot schema definition that was active during the session, and any tool-call or state-update events. The adapter serializes this into the [TRACE_DATA] placeholder, ensuring timestamps, turn indices, and slot names are preserved. The [SLOT_SCHEMA] placeholder should be populated from your application's form definition—the same schema the assistant was supposed to fill. This is critical: if the schema used for inspection differs from the schema active during the conversation, the audit will produce false positives. Model choice matters here. Use a model with strong structured output capabilities and a large context window (200K+ tokens) to accommodate long multi-turn traces. GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro are suitable. Avoid smaller models that may truncate mid-trace or hallucinate slot values when the trace is ambiguous.

Output validation is mandatory before trusting the audit. Parse the model's JSON response and validate it against the expected schema: every slot in [SLOT_SCHEMA] must appear in the audit output with a status (filled, missing, overwritten, contradictory, unconfirmed), the turn where the value was last set, and the final value. If the model's output fails schema validation, implement a single retry with the validation error appended to the prompt as additional context. If the retry also fails, flag the trace for human review rather than silently accepting a malformed audit. For high-stakes workflows—booking systems, healthcare intake, financial applications—route all audits with contradictory or overwritten slot statuses to a human review queue. These are the failure modes where the assistant silently discarded or altered user-provided information, and automated remediation is risky.

Logging and observability should capture: the trace ID, the model used, token consumption, validation pass/fail, the number of slots flagged per severity, and whether the audit was auto-accepted or escalated. Store the raw audit JSON alongside the trace for downstream trend analysis. If you're running this prompt across thousands of sessions, batch traces by session length and slot complexity to manage cost. Short sessions with few slots can use a cheaper model; long, complex sessions warrant the stronger model. Do not wire this prompt into a synchronous user-facing flow—the latency and cost are inappropriate for real-time use. Instead, run it as a periodic job (hourly or daily) against completed sessions, feeding results into your monitoring dashboards and alerting on threshold breaches: more than N sessions with contradictory slots in a time window triggers an investigation.

What to avoid: Do not use this prompt on incomplete or mid-session traces. The slot-state audit assumes the conversation has reached a terminal state; partial traces will generate false missing flags. Do not skip the schema validation step—model output drift on structured JSON is common, and an unvalidated audit is worse than no audit. Do not treat the model's severity classification as ground truth for automated remediation. The prompt identifies what went wrong; deciding what to do about it—re-prompt the user, restore a previous slot value, escalate to support—requires application logic and, in regulated domains, human judgment.