Conversation Gap Analysis Prompt for Missing Answers

This prompt is designed for offline conversation quality assurance and improvement workflows. Its primary job is to compare user questions against assistant responses across a complete, labeled transcript and identify turns where the assistant provided a non-answer, deflection, or partial response. The ideal user is an AI engineer, QA analyst, or product manager responsible for auditing chatbot performance at scale, measuring answer completeness, or generating a remediation backlog for gaps in assistant behavior. You need a full transcript with clear speaker labels (e.g., 'User:' and 'Assistant:') before using this prompt; it is not a real-time correction tool and will not fix a live conversation.

Use this prompt when you have a batch of completed conversations and need a structured gap report with answer completeness scores and suggested remediation. It is most effective as part of a pre-release regression testing suite or a periodic production audit pipeline. Do not use this prompt for real-time intervention, session state management, or direct user-facing correction. It assumes the conversation is finished and that the analysis can be performed asynchronously. For live systems that need to detect and recover from dropped questions mid-session, pair this analysis output with a state-tracking middleware and a separate 'Conversation Repair Prompt for Forgotten Questions' that acts on the detected gaps.

Before running this prompt, ensure your transcript is clean and consistently formatted. Inconsistent speaker labels, missing turns, or unmarked system messages will degrade the analysis. The prompt works best when combined with a validation layer that checks the output schema for required fields like turn_reference, completeness_score, and gap_type. For high-stakes audits where a missed gap could have compliance or safety implications, always route the generated gap report through a human review step before accepting remediation suggestions. The next step after reading this section is to copy the prompt template, adapt the [INPUT] and [OUTPUT_SCHEMA] placeholders to your transcript format, and wire it into an offline evaluation harness with schema validation and logging.

The following template is the core instruction set for the Conversation Gap Analysis prompt. It is designed to be dropped into an evaluation harness, a CI/CD pipeline, or a manual QA workflow. The prompt takes a full conversation transcript as input and returns a structured JSON report of every instance where a user's question was not fully answered. Use this as a starting point: replace the placeholder tokens with your actual transcript, output schema requirements, and any domain-specific constraints before deployment.

textCopy
Analyze the following conversation transcript between a User and an Assistant. For each user turn that contains a question or request for information, evaluate the assistant's subsequent response for answer completeness.

[TRANSCRIPT]

For each user question identified, produce a gap analysis entry with the following structure:
- turn_id: The turn number where the user asked the question.
- user_question: The exact question or request text.
- assistant_response_summary: A brief summary of what the assistant said in response.
- completeness_score: A score from 1 to 5 where 1 is a complete non-answer or deflection, 3 is a partial answer, and 5 is a fully complete and direct answer.
- gap_type: Classify the gap if score is 3 or below. Use one of: [DEFLECTION, NON_ANSWER, PARTIAL_ANSWER, IRRELEVANT_RESPONSE, ACKNOWLEDGMENT_ONLY, CLARIFICATION_LOOP].
- missing_information: What specific information was requested but not provided.
- suggested_remediation: A draft of what a complete answer should have included.
- severity: The impact of this gap on the conversation. Use [LOW, MEDIUM, HIGH, CRITICAL].

[OUTPUT_SCHEMA]
Return a JSON object with a "gaps" array containing one object per identified gap. If no gaps are found, return an empty array.

[CONSTRAINTS]
- Do not flag rhetorical questions or social pleasantries.
- If the user's question was answered across multiple assistant turns, evaluate the combined response.
- If the assistant asked a clarifying question and the user answered, evaluate the assistant's final response to the original question.

To adapt this template, start by replacing [TRANSCRIPT] with your actual dialogue data, ensuring each turn is clearly labeled with a speaker role and a turn number. The [OUTPUT_SCHEMA] placeholder should be replaced with a concrete JSON schema if your application requires strict validation, such as defining required fields and enum values. The [CONSTRAINTS] section is critical for tuning precision; for example, in a customer support context, you might add a constraint to ignore questions about order status if the assistant's response includes a tracking link, as that constitutes a complete answer in that domain. Always test the adapted prompt against a golden dataset of transcripts with known gaps to calibrate the completeness_score thresholds before production use.

This prompt is designed for offline batch processing, not a real-time endpoint. The analysis requires the full conversation transcript, which is only available after the session closes. Store transcripts in a database with a status field (active, closed, analyzed, reviewed). Run the gap analysis as an asynchronous job triggered by a conversation close event or a scheduled cron that picks up unprocessed closed transcripts. This decouples the analysis cost and latency from the user-facing assistant response time.

Parsing and validation are critical. The prompt requests a JSON output with a specific schema. Always parse the model response and validate each gap entry against the expected schema before storing results. If the model returns malformed JSON, use a repair prompt that feeds the raw output and the schema back to the model with stricter formatting instructions, or implement a retry with a lower temperature and explicit JSON mode. Do not silently accept partial or malformed gap reports. Validate that turn_reference values map to actual turns in the source transcript and that completeness_score falls within the expected 0.0–1.0 range.

Observability and drift detection should be built in from day one. Log the completeness_score distribution over time to detect systemic drift in assistant behavior—a downward trend may indicate a model update, prompt change, or knowledge base issue that is causing more non-answers. For high-severity gaps (e.g., severity: critical or completeness_score < 0.3), trigger a human review workflow by creating a ticket in your QA queue with the gap details and source transcript. Consider running this prompt on a daily sample of conversations rather than every conversation to manage cost, reserving full-coverage analysis for high-stakes or regulated use cases.