FastSpeech 2

Unlike autoregressive models like Tacotron 2 that generate output frames one at a time, FastSpeech 2 generates the entire mel-spectrogram sequence in a single, parallel forward pass. This architecture eliminates sequential dependencies, reducing latency from seconds to milliseconds. The core mechanism uses a feed-forward Transformer block that processes all phoneme embeddings simultaneously, followed by a length regulator that expands the sequence to the target mel-spectrogram length based on predicted durations.

FastSpeech 2 introduces three separate, lightweight predictor modules to model prosodic variations directly from the input, removing the need for a teacher model's approximations.

• Pitch Predictor: Models the fundamental frequency (F0) contour on a logarithmic scale, crucial for intonation and emotion.
• Energy Predictor: Captures the amplitude or loudness of speech over time.
• Duration Predictor: Estimates the number of mel-spectrogram frames each input phoneme should occupy. These predictors are trained with ground-truth extracted values, providing more accurate and stable prosody control than the previous knowledge distillation approach.

The Variance Adaptor is the central module that integrates the predictions from the pitch, energy, and duration predictors to condition the phoneme sequence. It performs three key operations:

1. Length Regulation: Expands the phoneme hidden sequence according to the predicted durations.
2. Pitch & Energy Embedding: Converts the continuous pitch and energy contours into embedding vectors.
3. Feature Addition: Adds the pitch and energy embeddings to the expanded phoneme sequence. This process injects rich, controllable prosodic information into the model before the final mel-spectrogram decoder, enabling fine-grained manipulation of speech style.

The decoder is based on a stack of Feed-Forward Transformer (FFT) blocks, a variant of the standard Transformer that replaces self-attention with a 1D convolutional network for local feature extraction, followed by a standard self-attention mechanism. This design is more efficient and stable for sequence generation tasks than recurrent networks. It takes the variance-adapted phoneme sequence and generates the final mel-spectrogram frames in parallel. The use of positional encoding ensures the model understands the temporal order of the output sequence.

FastSpeech 2 is trained using ground-truth targets extracted directly from the training audio, a significant shift from FastSpeech 1's reliance on knowledge distillation from an autoregressive teacher model. The model uses:

• Extracted Mel-Spectrograms as the generation target.
• Phoneme-level Durations from a forced aligner (e.g., Montreal Forced Aligner).
• Extracted Pitch (F0) and Energy values. This approach eliminates the teacher model's information bottleneck and exposure bias, improving training efficiency, stability, and the fidelity of the generated prosody.

By providing explicit control over the variance predictors, FastSpeech 2 enables precise, adjustable speech synthesis. At inference time, users can:

• Scale predicted pitch/energy values to make speech more expressive or monotone.
• Manually adjust phoneme durations to change speaking rate or add emphasis.
• Use reference audio to extract average pitch and energy values, transferring the global prosodic style to new text. This makes it highly suitable for applications requiring consistent voice characteristics with variable expressiveness, such as audiobooks, voice assistants, and emotional TTS systems.

A class of text-to-speech models that generate all output tokens (e.g., mel-spectrogram frames) in parallel, as opposed to sequentially. This is the core architectural paradigm of FastSpeech 2, enabling significant inference speedups (often 100x faster than autoregressive models like Tacotron) by eliminating sequential dependencies.

• Key Benefit: Drastically reduced latency for real-time applications.
• Challenge: Requires accurate upfront prediction of output sequence length (duration modeling).
• Example: FastSpeech 2, Parallel Tacotron.

The component within FastSpeech 2 responsible for predicting and injecting prosodic features into the speech. It uses separate, explicit predictors for:

• Duration: The number of mel-spectrogram frames per phoneme.
• Pitch (F0): The fundamental frequency, controlling vocal pitch.
• Energy: The amplitude of the signal, affecting loudness and emphasis.

These predicted variance features are added to the hidden phoneme sequence, providing fine-grained controllability over the speaking style and improving naturalness by modeling prosody separately from linguistic content.

A deep learning model that converts an intermediate acoustic representation (like a mel-spectrogram) into a raw audio waveform. FastSpeech 2 generates mel-spectrograms; a separate neural vocoder is required for the final synthesis.

• Function: Acts as a differentiable, learned inverse of the audio feature extractor.
• Examples: HiFi-GAN (efficient, GAN-based), WaveNet (high-quality, autoregressive), Diffusion-based vocoders.
• Importance: Vocoder quality is critical for the final audio fidelity and naturalness of any TTS pipeline.

The computational task of predicting and controlling the suprasegmental features of speech: rhythm (duration), stress, and intonation (pitch). Poor prosody leads to robotic, monotonous speech.

• FastSpeech 2's Approach: Uses explicit, frame-level predictors for pitch, energy, and duration, allowing for fine-grained control.
• Contrast with Tacotron: Earlier sequence-to-sequence models often learned prosody implicitly and could suffer from unstable attention or over-smoothing.
• Application: Essential for expressive TTS, emotional speech synthesis, and conveying meaning beyond the literal text.

A training methodology where a smaller, faster student model (FastSpeech 2) learns to mimic the output of a larger, more accurate teacher model. FastSpeech 1 & 2 used this to acquire alignment information.

• Process: An autoregressive teacher model (e.g., Transformer TTS) generates mel-spectrograms and its internal attention alignments.
• Use Case: The student model uses the teacher's alignments to derive duration labels, bypassing the need for a separate aligner and enabling parallel generation.
• Benefit: Allows the non-autoregressive student to achieve high quality while being far faster than the teacher.

A time-frequency representation of audio that is the primary intermediate target for most modern neural TTS systems, including FastSpeech 2. It is created by:

1. Computing the Short-Time Fourier Transform (STFT) of the waveform.
2. Mapping the frequency bins to the mel scale, which approximates human auditory perception.
3. Converting amplitude to decibels.

• Why it's used: It provides a compressed, perceptually-relevant representation that is easier for neural networks to generate than raw waveforms.
• Role in TTS: The TTS model (e.g., FastSpeech 2) predicts a mel-spectrogram sequence from text; the vocoder then converts this spectrogram back into a waveform.