Automatic Speech Recognition (ASR)

The Acoustic Model maps raw audio features to phonetic units or sub-word tokens. It learns the relationship between sound waves and the basic building blocks of speech.

• Input: Processed audio features like Mel-Frequency Cepstral Coefficients (MFCCs) or filter bank energies.
• Function: Models the probability of a phonetic unit given the acoustic signal, e.g., P(phoneme | audio frame).
• Modern Implementation: Typically a deep neural network, such as a Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), or Transformer, often trained with a Connectionist Temporal Classification (CTC) loss.

The Language Model predicts the probability of a sequence of words. It provides linguistic context to guide the decoder, distinguishing between acoustically similar phrases like "recognize speech" and "wreck a nice beach."

• Function: Models P(word sequence), capturing grammar, syntax, and common word associations.
• Types: N-gram models (statistical, now legacy) and Neural Language Models (based on RNNs or Transformers).
• Integration: Used during the decoding/search process to score and rank candidate word sequences generated by the acoustic model.

The Decoder is the search algorithm that finds the most probable word sequence given the acoustic input and language model. It combines scores from the acoustic and language models.

• Core Task: Find argmax P(Acoustic | Word Sequence) * P(Word Sequence).
• Common Algorithms: Beam Search is the standard, which explores a limited number of the most promising hypotheses (beams) at each time step.
• Modern Approach: End-to-end systems often use a beam search decoder over the combined output of a single neural model.

End-to-End ASR models collapse the traditional pipeline into a single neural network that directly maps audio to characters or words. They simplify architecture and can outperform modular systems.

• Architectures: Listen, Attend and Spell (LAS), Transformer Transducers, and RNN-Transducers (RNN-T).
• Advantage: Jointly optimizes all components, often leading to better performance with sufficient data.
• Output: Directly generates character or sub-word sequences, eliminating the need for a separate pronunciation dictionary.

The Feature Extraction frontend converts the raw audio waveform into a compact, informative representation suitable for the acoustic model. This step reduces dimensionality and highlights speech-relevant information.

• Common Features: Mel-Frequency Cepstral Coefficients (MFCCs), Filter Bank Energies, or Per-channel Energy Normalization (PCEN).
• Modern Trend: Deep neural networks (e.g., 1D Convolutions) can learn optimal feature representations directly from raw waveforms or spectrograms in an end-to-end fashion.

A Pronunciation Dictionary (or Lexicon) is a mapping between words and their possible phonetic pronunciations. It bridges the gap between the acoustic model's phonetic output and the language model's words.

• Function: Provides the sequence of phonemes for each vocabulary word (e.g., "cat" -> /k/ /æ/ /t/).
• Role in Pipeline: Used by the decoder to expand word hypotheses into phonetic sequences that can be scored by the acoustic model.
• Note: End-to-end systems that output characters directly do not require a pronunciation dictionary.

MFCCs are a handcrafted feature representation of the short-term power spectrum of a sound, crucial for traditional ASR pipelines. They are derived by:

• Applying a mel-scale filterbank to the audio's power spectrum to mimic human hearing.
• Taking the logarithm of the filterbank energies.
• Applying the Discrete Cosine Transform (DCT) to decorrelate the energies, producing the cepstral coefficients. While modern end-to-end ASR systems often use raw spectrograms or learn features directly, MFCCs remain a foundational concept in speech signal processing and are still used in resource-constrained environments.

CTC is a neural network output layer and alignment algorithm designed for sequence-to-sequence tasks where the input and output sequences are not explicitly aligned, such as speech-to-text. Its key mechanisms are:

• Introducing a blank token that allows the model to output a sequence longer than the target transcription.
• Using a dynamic programming algorithm (the forward-backward algorithm) to sum over all possible alignments during training, enabling learning without forced alignment.
• Collapsing repeated output tokens and removing blanks during decoding to produce the final transcription. CTC enables end-to-end training of ASR models without requiring pre-segmented audio data.

The RNN-Transducer is an end-to-end neural network architecture for streaming ASR that generalizes CTC. It consists of three components:

• An Encoder Network (e.g., a stack of RNNs or Transformers) that processes the acoustic input.
• A Prediction Network (an RNN) that models the language context of the output sequence so far.
• A Joint Network that combines encoder and prediction network outputs to produce a probability distribution over the next token. Unlike CTC, RNN-T has an internal language model, allowing it to use future linguistic context for more accurate, low-latency streaming transcription, making it a standard for on-device ASR.

Wav2Vec 2.0 is a framework for self-supervised pre-training of speech representations from raw audio. Its methodology involves:

• Masking spans of the raw audio waveform's latent representation.
• Training a context network (a Transformer) to identify the true quantized latent representation of the masked segment from distractors, a task called contrastive learning. This pre-training learns powerful, general-purpose acoustic representations without transcribed labels. The model is then fine-tuned on a small amount of labeled ASR data, achieving state-of-the-art performance with far less supervised data, revolutionizing data-efficient speech model development.

Whisper is a Transformer-based ASR model from OpenAI trained on 680,000 hours of multilingual and multitask supervised data. Its key architectural and training innovations include:

• A simple encoder-decoder Transformer trained to predict the transcript (or translation) of input audio.
• Multitask training on transcription, translation, voice activity detection, and language identification from a single model, controlled by special task tokens.
• Robustness to background noise, accents, and technical language due to the scale and diversity of its training data. Whisper is notable for its strong zero-shot performance across many languages and domains without fine-tuning, making it a versatile foundation model for speech.

This refers to the process of converting spoken audio into a dense vector embedding suitable for storage and retrieval in a multi-modal memory system (e.g., a vector database). The pipeline involves:

1. ASR Transcription: Converting speech to text using a model like Whisper.
2. Text Embedding: Using a language model (e.g., a Sentence Transformer) to encode the transcript into a semantic vector.
3. (Optional) Direct Audio Embedding: Using a model trained for audio retrieval (like a fine-tuned Wav2Vec 2.0) to create an embedding directly from the audio waveform, capturing paralinguistic features like tone and emotion. These embeddings enable semantic search over spoken content, allowing an agent to retrieve relevant past conversations or audio events based on meaning, not just keywords.