Voice Assistant Barge-In Recovery Prompt

This prompt is designed for a specific moment in a voice assistant pipeline: the system has detected a barge-in event, meaning the user began speaking while the assistant's text-to-speech (TTS) output was still playing. The core job is not to detect the interruption itself—your audio stack must already provide the barge-in signal and a partial ASR transcript of the overlapping speech. Instead, this prompt reconstructs what the user likely heard before they interrupted, extracts their new intent from the overlapping audio, and produces a structured recovery decision: continue the original response, restart it, or abandon it entirely. The ideal user is a developer or AI engineer integrating this prompt into a voice agent's turn-management logic, where losing context across an interruption leads to confusing, repetitive, or contradictory assistant behavior.

To use this prompt effectively, you must supply three critical inputs: the full text of the in-progress TTS response that was playing when the interruption occurred, the partial ASR transcript captured during the barge-in window, and a precise timestamp or word-index marker indicating how much of the TTS response had been spoken before the interruption. Without this temporal alignment, the model cannot reliably determine what the user heard versus what was queued but never vocalized. The prompt also expects a structured output schema, typically including fields for heard_segment (the reconstructed portion of the TTS response the user likely heard), barge_in_intent (the classified intent of the interrupting speech), recovery_action (one of continue, restart, or abandon), and rationale (a concise explanation linking the evidence to the decision).

Do not use this prompt when the interruption is a false positive—such as background noise, coughs, or cross-talk that does not contain a genuine user intent. It is also inappropriate for scenarios where the ASR transcript is empty or entirely unintelligible; in those cases, a simpler heuristic like 'ignore and continue' or 'reprompt the user' is more reliable and cost-effective. This prompt assumes the barge-in contains a meaningful user utterance that should change the assistant's behavior. If your system cannot distinguish between a true barge-in and ambient noise at the audio level, invest in that upstream classification before wiring in this recovery logic. Finally, avoid using this prompt for long-running agent tasks where the interruption invalidates multiple completed steps; that scenario requires a more complex state-checkpointing workflow covered by the Interrupted Agent Workflow Resume Prompt Template.

This template is designed to be injected into your voice assistant's dialogue manager immediately after the speech endpointing layer detects a barge-in event. It receives the assistant's interrupted text, the user's overlapping utterance, and the prior turn context. The prompt's job is not to generate a new response directly, but to produce a structured recovery decision that your application harness can act on: determining what the user likely heard, what they intended, and whether the prior response track is still viable. Replace every square-bracket placeholder with live data from your ASR output, text-to-speech buffer, and dialogue state store before sending this to the model.

textCopy
You are a voice assistant dialogue recovery module. Your task is to analyze a barge-in event where the user began speaking while the assistant was mid-response. You must reconstruct what the user likely heard, extract their barge-in intent, and recommend a recovery action.

## INPUTS

[PRIOR_TURN_SUMMARY]
Summary of the conversation before the assistant's interrupted response began. Include active intents, filled slots, and pending questions.

[ASSISTANT_RESPONSE_FULL_TEXT]
The complete text of the assistant response that was playing when the interruption occurred.

[ASSISTANT_WORDS_PLAYED_BEFORE_INTERRUPTION]
The prefix of the assistant response that was actually spoken before the user's speech overlapped. This is the portion the user likely heard.

[USER_BARGE_IN_UTTERANCE]
The user's overlapping speech, as transcribed by ASR. May be partial or noisy.

[USER_BARGE_IN_START_OFFSET_MS]
Milliseconds into the assistant response when the user began speaking.

[CONVERSATION_STATE_JSON]
Current dialogue state object including active slots, pending confirmations, and unresolved questions.

## OUTPUT SCHEMA

Return a valid JSON object with exactly these fields:

{
  "heard_prefix_summary": "String. What the user most likely heard before interrupting. Be conservative; note if the interruption point makes the heard content ambiguous.",
  "barge_in_intent": "String. Classify as one of: 'correction' (user is fixing something the assistant said or is about to say), 'answer_anticipation' (user is answering a question before it was fully asked), 'topic_shift' (user is changing the subject entirely), 'interjection' (user is adding a brief acknowledgment or filler), 'unclear' (insufficient signal to classify).",
  "intent_confidence": "Float between 0.0 and 1.0. Your confidence in the barge_in_intent classification.",
  "extracted_slots": "Object. Any slot values extractable from the user's barge-in utterance that update or add to the conversation state.",
  "recovery_action": "String. One of: 'continue' (the prior response track remains valid; resume from the interruption point or slightly before), 'restart' (the prior response is invalidated; generate a new response addressing the barge-in intent), 'abandon' (the prior response topic is no longer relevant; treat the barge-in as a new turn and discard the interrupted response), 'clarify' (insufficient information to decide; ask the user to repeat or clarify).",
  "restart_context": "String or null. If recovery_action is 'restart' or 'abandon', provide the key context the response generator needs. If 'continue' or 'clarify', set to null.",
  "requires_confirmation": "Boolean. True if the assistant should confirm its understanding with the user before proceeding, such as when extracted_slots are low-confidence or the barge-in intent is unclear.",
  "rationale": "String. Brief explanation of your recovery decision, referencing what the user heard and why the chosen action is appropriate."
}

## CONSTRAINTS

- Do not invent facts the user did not say. If the barge-in utterance is noisy or partial, reflect that in your confidence and prefer 'clarify' over guessing.
- If the user interrupted very early in the assistant response (first few words), they likely did not hear enough context. Consider whether the barge-in is a correction to the prior turn rather than the interrupted response.
- If the user's utterance is an acknowledgment ('ok', 'got it', 'thanks'), prefer 'continue' or 'abandon' based on whether the assistant was mid-information or mid-question.
- Preserve any confirmed slots from [CONVERSATION_STATE_JSON] unless the user's barge-in explicitly contradicts them.
- If recovery_action is 'continue', the response generator will need to know where to resume. Include enough detail in heard_prefix_summary to support that.

Adapt this template by tightening the barge_in_intent taxonomy to match your product's actual interruption patterns. If your voice assistant frequently encounters mid-question answers, add an answer_anticipation handler in your dialogue manager that skips the rest of the question and proceeds with the user's answer. If your ASR produces confidence scores, pass them alongside [USER_BARGE_IN_UTTERANCE] and add a [USER_ASR_CONFIDENCE] placeholder so the model can weigh transcription quality in its recovery decision. For high-stakes domains where incorrect slot extraction has real consequences, set requires_confirmation to true by default in your harness for any recovery action other than continue, overriding the model's judgment when safety matters more than latency.

The barge-in recovery prompt is not a standalone component; it sits inside a voice pipeline that must detect the interruption, capture the overlapping audio, transcribe the user's barge-in utterance, and then call this prompt to decide what to do next. The implementation harness must handle the full lifecycle: interruption detection from the voice activity detector (VAD) or stream endpoint, partial ASR on the overlapping segment, prompt assembly with pre-interruption context, and post-prompt action execution. Wire this prompt as a synchronous call after the ASR result for the barge-in segment is available, but before the system decides whether to replay, resume, or abandon the prior TTS output. The latency budget is tight—typically under 500ms end-to-end from end of barge-in speech to the system's next action—so keep the prompt concise and avoid unnecessary context bloat.

The harness requires four concrete inputs mapped to the prompt's placeholders. [PRIOR_RESPONSE_TEXT] is the exact text the system was speaking when the interruption occurred, truncated to the last N words actually sent to the TTS engine. [BARGE_IN_TRANSCRIPT] is the ASR output from the overlapping audio segment, which may be partial or noisy. [SESSION_CONTEXT] is a structured summary of the conversation state before the interruption, including active intents, filled slots, and pending actions. [INTERRUPTION_TIMING] captures the estimated word position or timestamp where the user began speaking relative to the prior response. Validate all four inputs before calling the model: if [BARGE_IN_TRANSCRIPT] is empty or below a confidence threshold, skip the prompt and execute a default clarification strategy (e.g., 'Sorry, I didn't catch that'). If [PRIOR_RESPONSE_TEXT] is missing, the prompt cannot reason about what the user heard, so fall back to treating the barge-in as a fresh turn.

Model choice matters. Use a fast, low-latency model for this prompt—GPT-4o-mini, Claude 3.5 Haiku, or Gemini 1.5 Flash are good candidates. Avoid large models that add hundreds of milliseconds of inference time. The output must be structured JSON with three fields: recovery_action (one of continue, restart, abandon, clarify), adjusted_response (the text to speak next, or null if abandoning), and rationale (a one-sentence explanation for logging). Validate the output schema strictly: if recovery_action is not in the allowed enum, default to clarify. If adjusted_response is provided for an abandon action, discard it. Log every prompt call with the inputs, outputs, latency, and the eventual user-facing action taken. This trace is essential for tuning interruption sensitivity and detecting false-positive barge-in triggers from background noise or self-talk.

Retries are rarely appropriate for this prompt because the latency cost of a second model call usually exceeds the user's patience. If the output fails schema validation, apply the default clarify action immediately rather than retrying. The exception is a malformed JSON response where a repair prompt can fix the structure in a single fast retry; implement a lightweight repair step that only corrects syntax, not semantics. For high-stakes voice applications—healthcare triage, financial transactions, emergency dispatch—add a human review flag when recovery_action is abandon and the prior response contained critical information. In these cases, log the full context for post-session review rather than silently dropping important content.

The prompt's output must feed directly into the TTS controller and dialogue state manager. If recovery_action is continue, send adjusted_response to TTS and update the dialogue state to reflect that the prior response was partially delivered. If restart, replay the full adjusted response from the beginning and reset the TTS cursor. If abandon, flush the TTS queue, acknowledge the user's interruption, and transition to processing the barge-in utterance as a new turn. If clarify, speak a clarification question and pause for user response. Wire the rationale field into your observability stack alongside VAD events, ASR confidence scores, and TTS playback timestamps. This end-to-end traceability is what separates a debuggable voice assistant from a black box that frustrates users with unpredictable interruption behavior.