Music Information Retrieval (MIR)

Automatic Music Transcription (AMT) is the process of converting an audio recording of music into a symbolic representation, such as a musical score or MIDI file. This involves identifying and notating:

• Pitch and onset times of individual notes.
• Duration and velocity (loudness) of notes.
• Polyphonic content (multiple simultaneous notes). It is considered a foundational but exceptionally challenging task due to the complexity of overlapping harmonics and percussive elements in polyphonic music. Applications include music education tools and archival of musical works.

Beat tracking is the task of identifying the periodic sequence of perceived pulse positions (the 'beat') in a musical audio signal. Tempo estimation (often measured in Beats Per Minute or BPM) determines the rate of this pulse. These tasks are critical for:

• DJ software and music synchronization.
• Music player visualizations.
• Dance video games and interactive systems. Algorithms typically analyze onset strength curves and use dynamic programming or probabilistic models to infer a consistent metrical grid, even in the presence of tempo changes (tempo rubato).

Key detection identifies the global tonal center (e.g., C major, A minor) of a musical piece. Chord recognition labels temporal segments with their underlying harmonic structure (e.g., C major, F minor seventh). These tasks rely on analyzing the distribution of pitch-class energy, often using:

• Chromagrams, a 12-dimensional representation showing the intensity of each musical pitch class over time.
• Hidden Markov Models (HMMs) or deep neural networks to model harmonic transitions. Accurate key and chord labeling is essential for music theory analysis, query-by-humming systems, and harmonic-aware music recommendation.

Music structure analysis (or segmentation) aims to decompose a song into its constituent, repeating sections, such as verse, chorus, bridge, and outro. The goal is to identify boundaries and group segments that are similar in musical content (melody, harmony, rhythm, timbre). Common approaches include:

• Computing a self-similarity matrix from audio features and detecting repetitive patterns.
• Using clustering algorithms on feature sequences. This structural understanding powers automatic summarization, thumbnailing, and intelligent navigation within long music collections.

This task involves computing a measure of perceptual or semantic similarity between music tracks to enable content-based music recommendation. It moves beyond collaborative filtering by analyzing acoustic properties directly. Systems extract high-level audio features or embeddings (e.g., from a neural network) to represent a track in a multidimensional space where proximity indicates similarity. Applications include:

• Playlist generation (e.g., 'radio' stations).
• Large-scale music library organization.
• Artist discovery platforms.

Instrument recognition (or timbre classification) identifies the presence or dominance of specific instruments (e.g., guitar, piano, drums) in an audio mixture. Source separation (or demixing) goes further by attempting to isolate the audio signal of individual instruments from the mixed recording. Key techniques include:

• Deep learning models like U-Nets trained on isolated stems.
• Non-negative Matrix Factorization (NMF) for simpler cases. These capabilities are vital for music education (learning a specific part), remixing, karaoke (vocal removal), and audio post-production.

MIR algorithms form the core of modern music streaming services like Spotify and Apple Music. By analyzing audio features such as tempo, key, timbre, and mood, these systems build rich acoustic profiles for every track. These profiles are used to:

• Power collaborative filtering and content-based recommendation.
• Generate personalized playlists (e.g., Discover Weekly, Daily Mixes).
• Create radio stations based on a seed song or artist. The goal is to model user taste and surface novel music by understanding deep sonic similarities, not just popularity or genre tags.

This application converts polyphonic audio (music with multiple instruments) into a symbolic notation like sheet music or a MIDI file. Key subtasks include:

• Multi-pitch detection: Identifying which notes are being played simultaneously.
• Onset detection: Pinpointing the precise start time of each note.
• Instrument recognition: Assigning detected notes to specific instruments (e.g., piano, guitar). This technology is used by musicians for learning songs, by archivists for digitizing recordings, and in music education software to provide real-time feedback.

Professional DJ software (e.g., Serato, rekordbox) and digital audio workstations (DAWs) rely heavily on MIR for automation and creative assistance.

• Beatgridding & Tempo Synchronization: Automatically detects the beats per minute (BPM) and aligns tracks for seamless mixing.
• Key Detection: Identifies the musical key of a track to suggest harmonically compatible songs for mixing.
• Structural Segmentation: Identifies verse, chorus, and bridge sections, allowing for intelligent looping and cue point setting. These features allow DJs and producers to focus on creativity rather than manual analysis.

Platforms like YouTube and SoundCloud use MIR for audio fingerprinting to manage copyright at scale. Systems like Shazam's core technology are based on MIR.

• Robust Hashing: Creates a unique, compact digital signature (fingerprint) for an audio file that is resilient to noise, compression, and equalization.
• Large-Scale Matching: Efficiently compares a query fingerprint against a database of millions of reference tracks to identify a song within seconds.
• Content ID: Continuously monitors uploaded content to detect copyrighted material, enabling rights holders to claim or monetize usage.

MIR enables large-scale computational analysis of music for academic and archival purposes.

• Trend Analysis: Studying the evolution of musical properties (e.g., loudness, harmonic complexity) over decades across genres.
• Cultural Musicology: Analyzing regional or genre-specific characteristics in global music corpora.
• Digital Library Navigation: Allowing researchers to search audio archives by acoustic similarity or specific musical attributes rather than just metadata. These tools provide data-driven insights into music history and culture that were previously infeasible through manual analysis.

MIR enables dynamic and responsive audio experiences in video games and interactive media.

• Adaptive Music Systems: Game soundtracks that change in real-time based on player action, using MIR to seamlessly transition between musical stems or adjust tempo and intensity.
• Procedural Audio Generation: Using analyzed musical features as parameters to generate new, context-appropriate music on the fly.
• Rhythm Game Scoring: Precisely aligning note charts to the detected beat and rhythmic structure of a song in games like Guitar Hero or Beat Saber. This creates a more immersive and personalized auditory experience.

A mel-spectrogram is a time-frequency representation of an audio signal critical for MIR feature extraction. It is created by applying a Short-Time Fourier Transform (STFT) and then warping the frequency axis to the mel scale, which approximates human auditory perception. This representation is the standard input for many deep learning models in MIR because it compresses the raw waveform into a perceptually relevant 2D image-like structure.

• Core Use: Input feature for convolutional neural networks (CNNs) in tasks like genre classification, chord recognition, and onset detection.
• Calculation: Computed by applying a mel filterbank to the power spectrogram, resulting in lower-dimensional, perceptually spaced frequency bins.

Symbolic music generation is the complementary task to MIR's analysis, focusing on creating musical compositions in discrete, structured formats like MIDI, MusicXML, or piano roll representations. Unlike raw audio generation, it deals with high-level musical constructs such as notes, chords, and tempo. MIR techniques are often used to analyze existing music to inform or train these generative models.

• Key Formats: MIDI sequences, ABC notation.
• Relationship to MIR: MIR systems can transcribe audio (audio-to-symbolic) to create training data for generative models, or evaluate the output of generative systems.

Timbre transfer is an audio synthesis task that changes the sound characteristics (timbre) of a source audio to match a target timbre, while preserving other musical content like pitch and rhythm. This relies heavily on MIR's ability to disentangle and represent different aspects of music. Techniques often use deep learning models to learn a latent space where timbre is encoded separately from other musical features.

• Example: Transforming a recording of a violin piece to sound as if played by a flute.
• MIR Foundation: Requires precise feature extraction to isolate timbre from melody and harmony, often using source separation or learned embeddings.

Onset detection is a fundamental low-level MIR task that identifies the precise starting points of musical notes or events in an audio signal. It is a prerequisite for higher-level analysis like beat tracking, tempo estimation, and automatic transcription. Algorithms typically look for sudden changes in energy or phase in the frequency domain.

• Methods: Spectral flux, phase deviation, complex domain analysis.
• Applications: The first step in building a piano roll representation from audio; essential for drum transcription and musical score alignment.

Beat tracking is the process of computationally determining the periodic sequence of perceived musical pulses (the beat) from an audio signal. Tempo estimation (measured in BPM - Beats Per Minute) is the related task of identifying the rate of this pulse. These are core MIR tasks that enable music synchronization, interactive systems, and structural analysis.

• Challenges: Handling tempo changes, variable note densities, and music without a strong percussive element.
• Algorithms: Often use autocorrelation of onset detection functions or probabilistic models like dynamic Bayesian networks.

Chord recognition (or chord estimation) is the MIR task of labeling temporal segments of music audio with their underlying harmonic structure, typically using chord symbols like C major or F# minor. It involves mapping complex audio spectra to a discrete vocabulary of chord labels. This is a high-level semantic task that combines signal processing with music theory.

• Approaches: Template matching with chroma features, hidden Markov models (HMMs), and deep learning classifiers (e.g., CNNs on chromagrams).
• Output: A sequence of chord labels aligned to the audio timeline, useful for music analysis, search, and interactive accompaniment systems.