Short-Time Fourier Transform (STFT)

The window function is applied to each segment of the signal before the Fourier Transform to minimize spectral leakage caused by discontinuities at the segment edges. The choice of window directly impacts the frequency resolution and sidelobe suppression of the resulting spectrogram.

• Common Windows: Hamming, Hann (Hanning), Blackman, and rectangular windows are standard.
• Trade-offs: A Hann window provides good frequency resolution and low spectral leakage, while a rectangular window offers superior time resolution but introduces significant spectral artifacts.
• Application: For speech analysis, a Hamming window is often used due to its good compromise between main lobe width and side lobe attenuation.

The window length (N) is the number of samples in each analysis segment. It is the most critical parameter, defining the fundamental trade-off between time and frequency resolution, governed by the Heisenberg-Gabor limit.

• Long Windows (> 20 ms): Provide high frequency resolution but poor time localization. Ideal for analyzing stationary signals like sustained musical notes.
• Short Windows (< 20 ms): Provide high time resolution but poor frequency discrimination. Essential for capturing transient events like speech plosives (e.g., /p/, /t/).
• Rule of Thumb: For speech processing, a window length of 25-30 ms (e.g., 400 samples at 16 kHz) is standard, as it roughly corresponds to the period over which the vocal tract is assumed to be stationary.

The hop length (or stride) is the number of samples the window is shifted between successive STFT calculations. It determines the overlap between consecutive frames and the temporal density of the spectrogram.

• Overlap Percentage: A hop length of N/2 results in a 50% overlap, N/4 results in 75% overlap.
• Purpose: Overlap is used to compensate for the attenuation of signal at the edges of the window function, ensuring all parts of the signal contribute equally to the analysis.
• Trade-off: Smaller hop lengths (higher overlap) produce smoother, more redundant time-frequency representations but increase computational cost. Larger hop lengths reduce cost but can cause temporal aliasing and miss rapid events.

The FFT size (N_FFT) is the number of points used to compute the Discrete Fourier Transform (DFT) via the Fast Fourier Transform (FFT) algorithm. It is typically equal to or greater than the window length (N).

• Zero-Padding: If N_FFT > N, the windowed segment is padded with zeros. This does not add new information but interpolates the frequency spectrum, providing a smoother visual representation in the spectrogram.
• Frequency Bins: The number of unique frequency bins in the output is (N_FFT/2) + 1. A larger N_FFT increases the number of frequency bins, offering finer granularity in the frequency axis.
• Performance: N_FFT is often chosen as a power of two (e.g., 512, 1024) to maximize the computational efficiency of the FFT algorithm.

The Heisenberg uncertainty principle (Gabor limit) formalizes the intrinsic trade-off between time resolution (Δt) and frequency resolution (Δf): Δt * Δf ≥ 1/(4π). This is a fundamental constraint in time-frequency analysis.

• Consequences: You cannot simultaneously have arbitrarily precise localization in both time and frequency. Improving one degrades the other.
• Engineering Implication: The choice of window length (N) is a direct application of this principle. A long window yields a narrow Δf (good frequency resolution) but a wide Δt (poor time resolution).
• Adaptive Strategies: For analyzing signals with both transients and sustained tones, advanced techniques like wavelet transforms or the Constant-Q Transform (CQT) are used, which vary resolution across frequencies.

The spectrogram is the two-dimensional visual representation generated by the STFT, plotting time on the x-axis, frequency on the y-axis, and intensity (often log-magnitude) as color or brightness.

• Magnitude Spectrogram: Shows the absolute value (amplitude) of the complex STFT output, representing signal energy. It is computed as np.abs(STFT).
• Phase Spectrogram: Shows the phase angle of the complex STFT output, which is crucial for perfect signal reconstruction via the inverse STFT (ISTFT).
• Log-Mel Spectrogram: A common preprocessing step for deep learning models involves converting the linear-frequency magnitude spectrogram to a mel scale and applying a logarithm, resulting in a perceptually relevant representation that compresses dynamic range.

A spectrogram is the visual plot of a signal's frequency spectrum over time, generated directly from the magnitude of the STFT. It is the primary output used for analysis and as input to deep learning models.

• Axes: Time (x-axis), Frequency (y-axis), and Intensity/Amplitude (color or brightness).
• Types: Narrowband (longer window, high frequency resolution) vs. Wideband (shorter window, high time resolution).
• Application: The standard 2D input for convolutional neural networks in audio tasks, treating time and frequency as spatial dimensions.

MFCCs are a compact, perceptually motivated feature set derived from an STFT-based spectrogram. They are the de facto standard for representing speech signals.

• Pipeline: STFT → Mel-scaled filter bank (simulates human hearing) → Logarithm → Discrete Cosine Transform (DCT).
• Output: A small set of coefficients (e.g., 13-40) representing the spectral envelope, discarding fine harmonic structure.
• Use Case: Dominant feature for traditional speech recognition, speaker identification, and audio classification before the deep learning era.

The Constant-Q Transform is an alternative time-frequency representation where frequency bins are spaced geometrically (like musical notes) rather than linearly (like the STFT).

• Key Difference: Uses a variable window length—longer windows for low frequencies (high resolution), shorter for high frequencies (high time resolution).
• Advantage: Provides a more musically relevant representation, offering better resolution at lower frequencies critical for pitch and harmony analysis.
• Trade-off: Computationally more complex than the standard FFT-based STFT.

The Wavelet Transform provides a multi-resolution time-frequency analysis using scalable, localized wavelets as basis functions, unlike the fixed-window STFT.

• Core Concept: Uses mother wavelets scaled (dilated) and translated to analyze signal features at different resolutions.
• Advantage over STFT: No fixed time-frequency resolution trade-off. Excellent at isolating short-duration, high-frequency events (transients) and long-duration, low-frequency trends.
• Application: Used in signal compression (JPEG2000), transient detection, and analyzing non-stationary signals with abrupt changes.

The Gabor Transform is essentially an STFT that uses a Gaussian window function. It is the specific case of the STFT that achieves the optimal joint time-frequency resolution under the Heisenberg-Gabor uncertainty principle.

• Window Function: Uses a Gaussian window, (g(t) = e^{-\alpha t^2}), which is its own Fourier transform.
• Property: Minimizes the product of time and frequency standard deviations, providing the best possible localization in both domains simultaneously.
• Significance: Forms the theoretical foundation for the STFT's resolution limits and is used in computer vision for texture analysis via Gabor filters.

The Audio Spectrogram Transformer is a modern deep learning model that applies a Vision Transformer architecture directly to a spectrogram, treating it as a sequence of patches.

• Input: A standard log-mel spectrogram (derived from STFT) is split into fixed-size patches.
• Architecture: Uses a standard Transformer encoder with multi-head self-attention to model global relationships across time and frequency.
• Impact: Demonstrates that convolution-free, attention-based models can achieve state-of-the-art results on audio classification, using the STFT's output as its foundational representation.