Automatic Speech Recognition (ASR)

Unlike traditional cascaded systems that separately optimize acoustic, pronunciation, and language models, end-to-end (E2E) ASR directly maps speech to text tokens using a single neural network. This eliminates the need for hand-crafted pronunciation lexicons and reduces engineering complexity.

• CTC-based: Connectionist Temporal Classification aligns input frames to output tokens
• RNN-Transducer: Jointly models acoustic and linguistic context for streaming
• Attention-based Encoder-Decoder: Uses cross-attention for non-streaming, high-accuracy transcription
• Example: DeepSpeech uses a CTC-trained recurrent architecture for direct speech-to-text mapping

ASR systems operate in two distinct modes that trade off latency for accuracy. Streaming ASR emits tokens as audio arrives, minimizing Time-to-First-Token (TTFT) for real-time applications. Batch ASR processes complete utterances with bidirectional context for maximum accuracy.

• Streaming latency target: < 500ms end-to-end for natural conversation
• Batch accuracy gain: 10-30% relative WER reduction vs. streaming
• Hybrid approaches: Two-pass systems use streaming for display, batch for finalization
• Example: Google's voice search uses a streaming RNN-T for live display, followed by a bidirectional rescoring pass

A language model (LM) scores candidate transcriptions based on linguistic plausibility, correcting acoustic ambiguities. In E2E systems, LMs are integrated via shallow fusion, deep fusion, or cold fusion to improve rare word recognition and domain adaptation.

• Shallow fusion: Log-linear interpolation of acoustic and LM scores during beam search
• Internal LM subtraction: Prevents over-counting of the E2E model's implicit LM
• Domain LMs: Fine-tuned on in-domain text corpora to boost specialized vocabulary
• Example: Medical ASR uses a domain LM trained on clinical notes to correctly transcribe drug names and anatomical terms

Preprocessing stages that segment continuous audio into speech regions and identify who spoke when. Voice Activity Detection (VAD) discriminates speech from silence and noise, while speaker diarization clusters speech segments by speaker identity using embedding vectors.

• VAD metrics: Frame-level accuracy > 95% in clean conditions
• Diarization pipeline: Segmentation → speaker embedding extraction (x-vectors, d-vectors) → clustering
• Overlap handling: Target-speaker VAD isolates primary speaker during crosstalk
• Example: Meeting transcription systems use diarization to produce speaker-labeled transcripts like "Speaker A: ..." and "Speaker B: ..."

Word Error Rate (WER) is the standard metric for ASR accuracy, calculated as the minimum edit distance between the reference and hypothesis transcriptions. It accounts for substitutions, deletions, and insertions normalized by the reference length.

• Formula: WER = (S + D + I) / N × 100%
• State-of-the-art: < 2% WER on LibriSpeech test-clean; 5-8% on conversational speech
• Limitations: WER treats all words equally; semantic WER and BLEU address meaning preservation
• Example: A 5% WER means 1 in 20 words is incorrect—acceptable for search, problematic for medical dictation

The human-computer interaction layer that enables users to control systems through spoken commands rather than graphical or text-based input. VUIs rely on ASR as the critical front-end component to convert acoustic signals into machine-readable text before downstream processing.

• Enables hands-free, eyes-free interaction for automotive, industrial, and accessibility applications
• Typically paired with text-to-speech (TTS) for bidirectional audio communication
• Requires robust wake word detection and endpointing to manage turn-taking

A subfield of NLP focused on machine reading comprehension, intent classification, and entity extraction. NLU consumes the text output produced by ASR and maps it to structured semantic representations that downstream systems can act upon.

• Performs slot filling to extract parameters like dates, locations, and product names from transcribed utterances
• Handles ambiguity introduced by ASR transcription errors through robust intent classification
• Often implemented as a pipeline: ASR → NLU → Dialogue Manager

A serving technique where generated tokens are transmitted to the client sequentially as they are produced, minimizing perceived latency. In ASR contexts, streaming inference enables real-time transcription where partial results appear before the speaker finishes an utterance.

• Critical for low-latency voice applications like live captioning and simultaneous interpretation
• Contrasts with batch mode, which waits for complete audio segments before processing
• Requires careful management of Time-to-First-Token (TTFT) to maintain conversational flow

The process of maintaining a structured representation of user goals, intents, and slots across multiple turns in a conversational AI interaction. DST consumes ASR and NLU outputs to update a belief state that persists throughout the dialogue.

• Tracks slot-value pairs (e.g., destination: "London", date: "Friday") across turns
• Handles corrections where users revise previously stated information
• Essential for task-oriented voice assistants that require multi-turn interactions to complete complex goals

The specific mechanism of delivering partial LLM outputs token-by-token over a persistent connection. In voice interfaces, token streaming from the ASR engine allows the system to begin NLU processing before the user finishes speaking, dramatically reducing end-to-end latency.

• Uses Server-Sent Events (SSE) or WebSocket connections for continuous delivery
• Enables progressive rendering of transcriptions in real-time captioning systems
• Pairs with endpoint detection to signal when the utterance is complete and final transcription can be committed

The system architecture component responsible for storing, summarizing, and recalling the history of a dialogue session. In voice-driven systems, conversational memory must account for ASR transcription errors that could corrupt stored context.

• Implements short-term memory for current session context and long-term memory for user preferences
• Uses conversation summarization to compress lengthy dialogues into dense representations
• Critical for maintaining coherence when ASR confidence scores are low and corrections are needed