Chroma Features

Chroma features map the entire frequency spectrum onto 12 pitch classes (C, C#, D, D#, E, F, F#, G, G#, A, A#, B), representing the 12 semitones of the equal-tempered scale. This mapping is octave-invariant, meaning all frequencies corresponding to the same pitch class (e.g., C4 at 261.63 Hz and C5 at 523.25 Hz) are aggregated into the same bin. This abstraction discards absolute pitch information to focus on harmonic content, making the representation robust to changes in key and instrumentation.

The extraction pipeline involves several deterministic signal processing steps:

• Short-Time Fourier Transform (STFT): Computes the time-frequency representation of the audio signal.
• Pitch Class Filtering: The magnitude spectrum is mapped to chroma bins using a filter bank. Common methods include:
  • Constant-Q Transform (CQT): Provides geometrically spaced frequency bins, offering better resolution for lower frequencies where musical pitch perception is more critical.
  • Log-Frequency Spectrogram: A spectrogram with logarithmically spaced frequency bins, approximating human auditory perception.
• Normalization: The 12-dimensional chroma vector for each time frame is often normalized (e.g., L2 norm) to emphasize the relative distribution of energy across pitch classes.

By collapsing octaves, chroma features capture the harmonic signature of an audio segment—the set of notes being played simultaneously. This makes them highly effective for:

• Chord Recognition: Identifying the underlying chord progression in music.
• Key Detection: Determining the tonal center or key of a piece.
• Melody Analysis: Tracking the sequence of predominant pitch classes over time, which corresponds to the melodic line, even when accompanied by harmony. The representation is sensitive to the root note and quality (major, minor) of chords, providing a compact descriptor for musical structure.

A key strength of chroma features is their engineered invariance to specific audio characteristics, which simplifies downstream machine learning tasks:

• Octave Invariance: As noted, pitch is folded into a single octave.
• Timbre Robustness: While not perfectly invariant, the projection onto pitch classes reduces the impact of an instrument's unique spectral envelope (timbre), focusing on the fundamental harmonic structure.
• Loudness Invariance: Normalization steps minimize the effect of overall signal energy or volume changes. These properties make chroma features a stable, high-level representation for comparing musical similarity across different performances or arrangements of the same piece.

Several refined versions of basic chroma features exist to address specific limitations:

• Chroma Energy Normalized (CENS): Applies additional smoothing over time and log-normalization of energy to further reduce the effects of dynamics and timbre.
• Harmonic Pitch Class Profile (HPCP): A more refined variant that uses a finer frequency resolution and can weight harmonics to improve accuracy for complex tones.
• Chroma Delta Features: The first-order temporal difference (derivative) of the chroma vector, which captures information about chord changes and melodic movement. These variants are selected based on the specific audio analysis task, such as cover song identification versus real-time chord transcription.

For robust audio identification, chroma features provide a compact, timbre-invariant signature.

• Shazam-like Systems: While full systems combine multiple features, chroma-based fingerprints are highly effective. They create a constellation of chroma peaks across time that is resilient to noise, compression artifacts, and equalization changes.
• Plagiarism Detection: In digital rights management, chroma representations are used to scan large audio databases for unauthorized samples or melodic similarities by comparing harmonic contours.
• The strength lies in its focus on harmonic content over spectral envelope, making matches possible across different recording qualities and arrangements.

Chroma features assist in deriving the rhythmic structure of music by revealing periodicities in harmonic change.

• Harmonic Rhythm: Chord changes often align with musical measures and beats. By analyzing the autocorrelation of the chroma vector over time, algorithms can detect the periodicity of these changes, which strongly correlates with the tempo.
• Downbeat Detection: The transition to a new chord or a strong harmonic event frequently coincides with the downbeat (first beat of a measure). Chroma flux (the rate of change between chroma vectors) is a key signal for identifying these structural boundaries.
• This use case demonstrates chroma's role in mid-level feature extraction, bridging low-level audio signals to high-level musical structure.

Chroma features enable the automatic decomposition of a song into its constituent parts (verse, chorus, bridge, etc.).

• Self-Similarity Matrices: A common technique involves computing a chroma self-similarity matrix, where each point represents the similarity between the chroma vectors at two different times. Repeated sections (like choruses) appear as bright blocks along the diagonal, allowing for automatic segmentation.
• Boundary Detection: Sudden changes in the overall harmonic content, visible as discontinuities in the chroma sequence, often signal transitions between musical sections.
• This structural analysis is fundamental for music recommendation, automatic summarization, and interactive music applications.

While not as precise as dedicated pitch tracking algorithms for monophonic signals, chroma features provide a robust basis for polyphonic pitch analysis.

• In polyphonic music (multiple notes sounding simultaneously), the chroma vector represents the composite harmonic energy. Advanced models can deconvolve this to estimate the set of active pitches at each time frame.
• Piano Roll Representation: The output of chroma-based transcription is often visualized as a piano roll, a time-pitch grid showing when each of the 12 pitch classes is active, forming a foundational step towards full Automatic Music Transcription (AMT).
• This application highlights chroma's role in reducing the complex audio spectrum to a manageable, musically meaningful representation for downstream neural networks.

Chroma features serve as a powerful, domain-specific input representation for deep neural networks in audio AI.

• Input Conditioning: Instead of feeding raw audio or spectrograms, models for music classification or generation are often conditioned on chroma vectors to explicitly provide harmonic context, reducing the learning burden.
• Multi-Task Learning: In architectures like MusicBERT or other audio transformers, chroma can be used as an auxiliary input stream alongside Mel-spectrograms, providing a clear signal for harmonic tasks while the network learns timbral features from other inputs.
• Data Efficiency: By providing a strong musical prior, chroma features can lead to more data-efficient training and improved model performance on harmony-centric tasks compared to models learning purely from raw data.

The Constant-Q Transform (CQT) is the mathematical foundation for calculating chroma features. Unlike the Short-Time Fourier Transform (STFT), which uses linearly spaced frequency bins, the CQT uses geometrically spaced bins, providing higher frequency resolution at lower frequencies. This aligns with human perception and the musical scale, making it ideal for deriving the 12 chroma bins that represent the semitones of an octave. The CQT's output is a time-frequency representation where each column can be summed across octaves to produce the final chroma vector.

Mel-Frequency Cepstral Coefficients (MFCCs) are another fundamental audio feature, but they serve a different purpose than chroma. While chroma captures harmonic and melodic content, MFCCs are designed to represent the short-term power spectrum of sound, optimized for speech recognition. Key differences:

• Scale: MFCCs use a perceptually motivated mel scale, while chroma uses a musical scale.
• Focus: MFCCs emphasize spectral envelope (vocal tract shape), chroma emphasizes pitch class.
• Dimensionality: MFCCs typically produce 13-40 coefficients; chroma produces a fixed 12 dimensions. They are often used together in multimodal audio pipelines for comprehensive feature representation.

Pitch Class Profile (PCP) is a direct synonym for chroma features. The term emphasizes the core concept: mapping all spectral energy onto the 12 pitch classes (C, C#, D, ..., B) that repeat across octaves. The calculation involves:

• Computing the magnitude spectrum (via STFT or CQT).
• Mapping each frequency bin to its corresponding pitch class.
• Summing the energy within each of the 12 bins, often with octave normalization. This results in a robust representation that is invariant to changes in timbre and instrumentation, making it powerful for chord recognition and audio fingerprinting.

Harmonic-Percussive Source Separation (HPSS) is a crucial preprocessing step for clean chroma extraction. It decomposes an audio signal into two components:

• Harmonic Component: The part with stable pitch (e.g., melody, chords).
• Percussive Component: The transient, noise-like part (e.g., drums, hits). Applying chroma features to the raw audio can be noisy due to percussive energy. By first isolating the harmonic component using HPSS (often via median filtering on spectrograms), the resulting chroma vectors become significantly more accurate for tasks like chord detection and key estimation, as they reflect purely pitched content.

A chromagram is the visual or data structure representation of chroma features over time. It is a 2D matrix where:

• Rows: Represent the 12 pitch classes (chroma).
• Columns: Represent successive time frames.
• Cell Values: Represent the intensity of a given pitch class at a given time. Chromagrams are the standard input format for downstream machine learning models in Music Information Retrieval (MIR). They enable the analysis of harmonic progression, key changes, and musical structure. For a 3-minute song sampled at 10 frames per second, a chromagram would be a matrix of shape [12, 1800].

Tonal Centroid Features (or Tonnetz features) are an advanced representation derived from chroma vectors. They project the 12-dimensional chroma space into a 6-dimensional space representing harmonic relationships. This projection is based on the theory of the Tonnetz, a lattice that maps musical pitches based on perfect fifths and major thirds. The transformation highlights the circle of fifths and chordal relationships, making these features particularly effective for complex harmonic analysis where simple chroma may be ambiguous. They provide a geometrically meaningful embedding for chord classification and modulation detection.