Tacotron 2

The first stage of Tacotron 2 is a sequence-to-sequence model with an attention mechanism that predicts a sequence of mel-spectrogram frames from an input sequence of characters or phonemes.

• Encoder: A stack of convolutional layers followed by a bidirectional LSTM converts the text into a hidden feature representation.
• Location-Sensitive Attention: This mechanism aligns the encoder states with the target mel-spectrogram frames, allowing the model to learn which parts of the text correspond to which parts of the audio.
• Decoder: An autoregressive LSTM predicts one mel-spectrogram frame at a time, conditioned on the previous frame and the attended encoder context.

This architecture enables the model to learn complex, long-range dependencies between text and acoustic features, producing highly natural prosody.

Instead of using a traditional signal processing vocoder (like Griffin-Lim), Tacotron 2 employs a modified WaveNet as a neural vocoder to generate the final time-domain waveform.

• Input Conditioning: The predicted mel-spectrograms from the first stage are used to condition the WaveNet, guiding it to generate the corresponding raw audio samples.
• Autoregressive Generation: WaveNet generates audio sample-by-sample, with each new sample's probability distribution conditioned on all previously generated samples, capturing fine-grained waveform details.
• Dilated Causal Convolutions: These allow the model to have a very large receptive field, enabling it to model long-term dependencies in audio crucial for natural-sounding speech.

This combination results in audio quality that approaches human-recorded speech, with reduced buzziness and artifacts common in older systems.

Tacotron 2 is trained in two separate, independent stages, which simplifies the training process and improves stability.

• Stage 1: Spectrogram Prediction: The sequence-to-sequence model is trained using a simple L1 loss between its predicted and ground-truth mel-spectrograms. This stage learns the mapping from text to acoustic features.
• Stage 2: Vocoder Training: The WaveNet vocoder is trained separately to maximize the likelihood of real waveform samples, conditioned on their corresponding ground-truth mel-spectrograms. This teaches it to invert the mel-spectrogram representation back into high-fidelity audio.
• Inference Coupling: During inference, the two trained components are connected: the first stage's output is fed directly into the second. This modularity allows for potential component swapping (e.g., using a different, faster vocoder like HiFi-GAN).

Tacotron 2 operates directly on raw character sequences, eliminating the need for a separate, complex front-end text normalization system or linguistic feature extractor.

• Embedding Layer: An initial embedding layer converts the input characters into continuous vectors.
• Convolutional Pre-Net: A small stack of convolutional layers processes these embeddings, capturing local patterns like n-grams and simple phonemic context before the encoder LSTM.
• Implicit Learning: The model learns grapheme-to-phoneme relationships, punctuation effects, and prosodic patterns directly from data. For example, it learns that a question mark often implies a rising intonation.

This end-to-end approach simplifies the pipeline and improves robustness across different languages and writing styles.

The primary achievement of Tacotron 2 was producing speech with unprecedented naturalness and audio fidelity, as measured by subjective Mean Opinion Score (MOS) tests.

• Prosody: The combination of the attention-based decoder and autoregressive WaveNet captures the nuanced rhythm, stress, and intonation of human speech far better than previous parametric TTS models.
• Spectral Detail: The WaveNet vocoder generates waveforms with rich harmonic structure and crisp consonants, avoiding the muffled or metallic sound of traditional vocoders.
• Benchmark Performance: In initial evaluations, Tacotron 2 achieved a MOS close to that of professional studio recordings, significantly outperforming its predecessor, Tacotron, and contemporary parametric systems.

This quality established a new standard for neural TTS and directly influenced all subsequent architectures.

While groundbreaking, Tacotron 2 has inherent limitations that motivated further research, placing it in a clear evolutionary context.

• Inference Speed: The autoregressive nature of both its decoder and WaveNet vocoder makes generation slow, as outputs must be produced sequentially.
• Controllability: Prosodic features like pitch and speaking rate are learned implicitly but are not easily disentangled for fine-grained, explicit control during synthesis.
• Robustness: The attention mechanism can sometimes produce alignment failures (repetitions or skips) on unusual or out-of-distribution text inputs.

These limitations led to the development of non-autoregressive models like FastSpeech 2 (for parallel spectrogram prediction) and GAN-based vocoders like HiFi-GAN (for parallel waveform generation), which trade some initial quality for massive speed improvements.

A mel-spectrogram is a time-frequency representation of an audio signal where the frequency axis is warped to the mel scale, which approximates human auditory perception. It is the primary intermediate acoustic feature used by Tacotron 2's sequence-to-sequence predictor.

• Purpose: Serves as a compressed, perceptually-relevant target for the first stage of a two-stage TTS pipeline.
• Calculation: Derived by applying a Short-Time Fourier Transform (STFT), then mapping the frequencies to the mel scale using a filter bank.
• Role in Tacotron 2: The model's encoder-attention-decoder network predicts a sequence of mel-spectrogram frames from input text. This representation is then fed to the WaveNet vocoder.

A neural vocoder is a deep learning model that generates raw audio waveforms from intermediate acoustic representations like mel-spectrograms. Tacotron 2 uses a WaveNet vocoder for this final synthesis step.

• Core Function: Converts low-frequency, frame-based features (e.g., 80-band mel-spectrograms at ~12.5 ms frames) into high-frequency audio samples (e.g., 24 kHz waveform).
• Architectures: Includes autoregressive models (WaveNet), Generative Adversarial Networks (HiFi-GAN), and diffusion-based models (WaveGrad, DiffWave).
• Impact: The shift to neural vocoders from traditional signal-processing methods (like Griffin-Lim) was the single largest factor in achieving human-like speech quality in systems like Tacotron 2.

WaveNet is a deep autoregressive neural network architecture for generating high-fidelity raw audio waveforms, developed by DeepMind. It serves as the vocoder in the original Tacotron 2 paper.

• Mechanism: Models the conditional probability of each audio sample given all previous samples and a conditioning mel-spectrogram, using dilated causal convolutions.
• Key Innovation: Its dilated convolutions provide an exponentially large receptive field, allowing it to capture long-range dependencies in audio crucial for coherence and naturalness.
• Limitation & Evolution: Its autoregressive nature makes it computationally slow for inference. This led to the development of parallel alternatives like HiFi-GAN and Parallel WaveNet.

FastSpeech 2 is a non-autoregressive text-to-speech model that generates mel-spectrograms in parallel, addressing key speed and controllability limitations of autoregressive models like Tacotron 2.

• Core Contrast to Tacotron 2: Eliminates the sequential decoder, enabling parallel mel-spectrogram generation for vastly faster inference.
• Explicit Variance Prediction: Uses separate predictors for pitch, energy, and duration (prosodic features), providing finer-grained, more stable control over speech characteristics than Tacotron 2's implicit learning.
• Trade-off: While faster and more controllable, later iterations of FastSpeech often require a separately trained model (like a GAN) to achieve the same peak audio quality as the Tacotron 2 + WaveNet combination.

Prosody modeling is the computational task of predicting and controlling the rhythm, stress, and intonation of synthesized speech. Tacotron 2 learns prosody implicitly from data via its attention mechanism.

• Elements: Includes duration (phone and word timing), pitch (fundamental frequency contour), and energy (loudness or amplitude).
• Challenge in Tacotron 2: Prosody is entangled in the decoder's latent states, making targeted control difficult. Instability in the attention mechanism can also lead to mispronunciations or erratic rhythm.
• Modern Approach: Subsequent models like FastSpeech 2 and VITS disentangle prosody by using explicit, externally predicted variance features (pitch, energy, duration) as additional inputs to the decoder.

HiFi-GAN is a generative adversarial network-based neural vocoder designed for efficient and high-fidelity waveform generation from mel-spectrograms. It is a modern, faster successor to the WaveNet vocoder used in Tacotron 2.

• Architecture: Uses a generator with transposed convolutions and multiple discriminators (Multi-Scale and Multi-Period Discriminators) to assess audio quality at different scales.
• Advantages over WaveNet:
  • Non-autoregressive: Generates the entire waveform in parallel, enabling real-time synthesis on a single GPU.
  • High Quality: Achieves Mean Opinion Scores (MOS) comparable to or exceeding autoregressive vocoders.
• Practical Use: Often used as a drop-in replacement for WaveNet in Tacotron 2-style systems to dramatically improve inference speed.