How to Clean Audio for an AI Music Dataset

Audio collected from different sources rarely arrives in one neat format. One track may be stereo at 44.1 kHz, another mono at 48 kHz, and another stored in a damaged container. Some files begin with long silence. Others are clipped, unusually quiet, repeated under different names, or several hours longer than the model can process at once.

Cleaning turns this mixed collection into reliable training examples. The goal is not to make every recording sound identical. The goal is to make the files technically compatible, remove records that cannot be trusted, and keep useful musical differences intact.

A strong workflow also keeps the original files. Cleaning should create traceable derivatives rather than silently replacing the source material.

A practical audio preparation pipeline

Keep the original
Store the source file and its provenance before creating any processed version.
Inspect and decode
Check that the container and audio stream can be read, then record duration, codec, sample rate, channels, and other technical properties.
Standardise the signal
Convert the audio to the sample rate, channel layout, and numerical format expected by the model.
Measure quality
Check loudness, true peak, silence, clipping, duration, and unusual values before deciding whether to change or reject the record.
Segment the audio
Create examples that fit the model's context while keeping captions, lyrics, MIDI, and timestamps aligned.
Find duplicates
Detect exact and near-identical recordings before assigning records to dataset splits.
Save the processing history
Record every change so the prepared example can be traced back to the original.

1. Validate before converting

A file extension does not prove that a file contains valid audio. A file named .wav may be incomplete, use an unsupported codec, or contain no usable audio stream.

A validation step should inspect the file before expensive processing begins. Useful checks include:

Can the container be opened?
Is there at least one usable audio stream?
Is the codec supported by the pipeline?
Are sample rate, channel count, and duration present and sensible?
Can the audio be decoded without a fatal error?
Does the decoded signal contain finite values rather than NaN or infinity?

FFprobe can report stream and format information in a machine-readable form. The FFmpeg documentation also provides a stream selector for streams with usable configuration, which requires essential information such as the audio sample rate.

inspect-audio.sh · A simple FFprobe command for inspecting the first audio stream.

Swipe sideways to view code

ffprobe -v error \
  -select_streams a:0 \
  -show_entries stream=codec_name,sample_rate,channels,channel_layout,duration \
  -show_entries format=format_name,duration,size \
  -of json input.wav

2. Resample to the model's expected rate

Models normally expect one sample rate. If the model was designed for 32 kHz audio, feeding it a mixture of 22.05, 44.1, and 48 kHz files would give the same number of samples different meanings across records.

Resampling converts the signal from its original sample rate to the target rate. It does not simply discard samples at regular intervals. A proper resampler filters and interpolates the signal so frequencies above the new Nyquist limit do not fold back into the audible range as aliasing.

PyTorch's official audio tutorial provides bandlimited resampling functions and lets developers control filter width and roll-off. The same target rate should be used consistently during training and inference.

3. Standardise the channel layout

Mono and stereo examples have different shapes. A model must either support both layouts or receive a consistent conversion.

Downmixing stereo to mono reduces the amount of audio data and removes the need to model left-right placement. It also removes stereo width and can change the balance of sounds that differ between the channels. Converting mono to stereo by copying one channel does not create real spatial information.

The choice should match the model. MusicGen's main training setup downmixed audio to mono unless a stereo experiment was being run. Stable Audio Open was built for stereo output. Keep the original channel layout in metadata even when the training derivative uses a different one.

What should be standardised?

Swipe sideways to view the full comparison

Property	Why consistency helps	What may be lost or changed
Sample rate	Each sample has the same time scale and frequency limit	Very high frequencies may be removed when downsampling
Channel layout	Batches have a consistent shape	Downmixing can remove stereo placement
Numerical type	The model receives a predictable value range and precision	Poor conversion can clip or quantize the signal
Segment length	Examples fit the model context and batch	Bad boundaries can cut phrases, words, or labels
Loudness policy	Extreme level differences can be controlled	Aggressive normalisation can remove useful level variation

4. Measure loudness before changing it

Amplitude peaks and perceived loudness are not the same thing. ITU-R BS.1770 defines algorithms for measuring programme loudness and true-peak level. Loudness measurement uses frequency weighting, channel weighting, and gating to estimate how loud a programme may feel.

A dataset may use loudness normalisation to stop extremely quiet files from contributing tiny signals and to reduce large level differences between sources. Stability AI's training tools include optional normalisation to a target LUFS value, followed by optional gain variation.

Normalisation should not be automatic in every project. Loudness may carry useful information for dynamics, expressive performance, sound effects, or mastering style. A good pipeline measures first, sets an allowed range, and only changes records that fall outside the policy.

Peak level and loudness are different checks

Swipe sideways to view the full comparison

Measurement	What it describes	Why it matters
Sample peak	The largest stored sample value	Can reveal obvious digital clipping risk
True peak	An estimate of peaks that can occur between stored samples	Helps measure remaining headroom more accurately
Integrated loudness	An estimate of programme loudness across time	Supports more consistent level policies across recordings

5. Treat silence as data, not just empty space

Long silence can waste model capacity, especially when a random crop lands in a quiet section. Librosa provides functions for trimming leading and trailing silence and splitting a signal into non-silent intervals using a threshold relative to a reference level.

Stability AI's dataset pipeline can optionally strip trailing silence, shorten long internal silent sections, skip pure-silence samples, and produce a padding mask showing which frames contain real audio.

Still, silence can be musically meaningful. A pause before a chorus, space between notes, or a quiet introduction is part of the composition. Remove only silence that is clearly unusable for the task. Save thresholds and masks so the decision can be reviewed.

Possible ways to handle silence

Swipe sideways to view the full comparison

Action	When it may help	Risk
Keep it	Pauses and dynamics matter to the task	Long empty crops may waste training steps
Trim the beginning or end	The silence comes from recording or export padding	A quiet entrance or decay may be cut
Shorten long internal gaps	The dataset contains unusable dead air	Song timing and structure may be changed
Skip pure-silence segments	The crop contains no meaningful signal	Very quiet music may be rejected by a poor threshold
Keep a padding mask	Short files must be padded to a fixed length	The model and loss must use the mask correctly

6. Turn long tracks into model-sized examples

A full track is often longer than the model's training context. Segmentation divides it into shorter examples.

MusicGen trained on random 30-second crops sampled from full tracks. This gave the model a fixed amount of audio per example while exposing it to different parts of each recording across training.

Random cropping is simple, but it is not correct for every task. A caption may describe the whole song but not a specific crop. Lyrics, MIDI, and structural labels must be cut and shifted to match the selected time range. For transcription or section modelling, boundaries based on notes, phrases, bars, or annotated sections may be more useful than arbitrary cuts.

Ways to create segments

Swipe sideways to view the full comparison

Strategy	Strength	Main risk
Fixed non-overlapping windows	Simple and reproducible	Boundaries can cut musical events
Overlapping windows	Reduces boundary loss	Creates highly similar examples
Random crops	Provides variation across epochs	A condition may not describe every crop
Silence-based boundaries	Can follow natural gaps	Quiet music may be split incorrectly
Musical or annotated boundaries	Can preserve phrases, bars, lyrics, or sections	Requires reliable analysis or annotations

7. Find duplicates before splitting the dataset

Duplicates are not limited to files with the same name. The same recording may appear as WAV and MP3, with different loudness, shortened silence, changed metadata, or a slightly different duration.

Use several levels of checking:

A file hash finds byte-for-byte copies.
Basic metadata and duration can narrow the search.
An acoustic fingerprint can identify near-identical audio after re-encoding or small changes.
Similarity models can create a review list for harder cases, but similarity alone should not automatically prove that two tracks are duplicates.

Chromaprint is designed for full-file identification and duplicate audio detection. It aims to identify near-identical audio rather than broad musical similarity.

In practice

Real example: duplicates changed the benchmark

Researchers studying the SoundDesc audio-text retrieval dataset found duplicate recordings across the training and evaluation data. The overlap produced overly optimistic retrieval results. They created revised splits and grouped files that shared similar recording setups to reduce contamination.

In practice

Implementation example: Stable Audio Tools

Stability AI's open training tools document a raw-audio pipeline that loads and resamples audio, optionally normalises loudness, strips trailing silence, removes long internal silence, pads or crops to the model sample size, skips pure-silence samples, and converts the channel layout. It also stores a padding mask and normalised timing metadata. This is a useful example of how separate cleaning decisions become one repeatable pipeline.

processed-audio-record.json · A simplified processing record that keeps the derivative traceable.

Swipe sideways to view code

{
  "sourceId": "track-0042-original",
  "sourcePath": "originals/track-0042.flac",
  "processedPath": "training/track-0042__0030-0060.wav",
  "sourceSampleRateHz": 44100,
  "targetSampleRateHz": 32000,
  "sourceChannels": 2,
  "targetChannels": 1,
  "segmentStartSeconds": 30,
  "segmentDurationSeconds": 30,
  "integratedLoudnessLufs": -18.7,
  "gainAppliedDb": 0,
  "silenceAction": "none",
  "duplicateGroupId": "recording-group-184",
  "processingVersion": "audio-clean-v1"
}

Interactive lesson

Clean a Training Clip

Try it: Begin with validation and inspect the technical report. Apply resampling, channel conversion, silence handling, and segmentation one step at a time. Finish by reviewing the simulated audio fingerprint match and assigning the source group to a dataset split.

A reliable audio pipeline validates and decodes each source, converts it to the model's expected sample rate and channel layout, measures loudness and true peak, handles silence carefully, creates aligned segments, detects duplicates before splitting, and records every processing step.

Collecting and Cleaning Audio

A practical audio preparation pipeline

Keep the original

Inspect and decode

Standardise the signal

Measure quality

Segment the audio

Find duplicates

Save the processing history

1. Validate before converting

2. Resample to the model's expected rate

3. Standardise the channel layout

4. Measure loudness before changing it

5. Treat silence as data, not just empty space

6. Turn long tracks into model-sized examples

7. Find duplicates before splitting the dataset

Real example: duplicates changed the benchmark

Implementation example: Stable Audio Tools

Clean a Training Clip

Ready for a quick check?

Ready to continue?