Learn how tags, metadata, and natural-language captions describe music, why labels often disagree, and what makes a caption useful for model training.
About 11 minutes
Guiding question
What makes a caption useful for conditioning a music model?
By the end, you’ll be able to
Explain the difference between metadata, tags, and natural-language captions.
Write a useful music caption based on audible evidence.
Identify labels that are vague, inconsistent, subjective, or unsupported.
Explain how annotation quality affects text-conditioned music models.
A model does not automatically know that a clip contains slap bass, a fast drum groove, or a relaxed mood. It learns these relationships from words attached to audio.
Those words can take several forms. A catalogue record may contain the artist, release year, and title. A tagging dataset may attach labels such as funk, bass guitar, and energetic. A caption may describe the clip in a full sentence.
These fields are related, but they are not interchangeable. Good metadata helps organise a collection. Good tags provide consistent categories. Good captions explain what can actually be heard. A useful training record keeps those roles clear.
Three language layers in a music record
Swipe sideways to view the full comparison
Layer
Typical content
Main strength
Main risk
Catalogue metadata
Artist, title, album, year, language, source, and licence
Identifies and organises the recording
May describe the release rather than the selected audio segment
Tags
Genre, instrument, mood, voice, or sound-event labels
Easy to filter, count, and use as model targets
Labels can be broad, inconsistent, missing, or subjective
Caption
A sentence or paragraph describing what is heard
Can express several musical ideas and relationships together
Can become vague or invent details that are not supported by the audio
Tags: short and structured
Tags are useful when the dataset needs a fixed set of categories. A record can contain several tags at once because music rarely belongs to only one simple class.
MTG-Jamendo contains tags from genre, instrument, and mood or theme categories. AudioSet uses a larger ontology of sound-event labels that includes musical instruments, singing, and music genres alongside many non-musical sounds.
Tags make counting and evaluation easier. A model can learn whether a clip contains guitar, vocals, or percussion without first learning to interpret a full sentence.
The trade-off is that a tag says little about relationships. The labels piano, female vocal, and slow do not explain whether the piano leads the song, whether the voice is soft or powerful, or how the elements interact.
Captions: richer musical descriptions
A caption can join several properties into one readable description. It can describe instrumentation, rhythm, tempo, vocal delivery, mood, production, and how the parts relate.
MusicCaps was created for text-to-music evaluation. It contains 5,521 ten-second music examples, each with an English aspect list and a free-text caption written by musicians. The MusicLM paper reports that the captions contain about four sentences on average, while the aspect lists contain about eleven musical aspects per clip.
This level of detail gives a text-conditioned model more guidance than a track title or one genre tag. It also creates more ways for a caption to be wrong.
In practice
Example: MusicCaps
MusicCaps keeps both structured aspects and free-form language. A record may list separate ideas such as pop, piano, high-pitched female vocals, and synthesizer, then express them together in a longer caption. The aspect list is easier to count, while the caption gives a model a more natural conditioning signal.
What makes a useful caption?
A strong caption gives the model information that is both useful and supported.
Useful means the description contains musical detail that can guide search or generation. Supported means the detail can reasonably be heard in the selected clip or verified from trusted record data.
The goal is not to write the longest possible sentence. A long caption with several incorrect details can be worse than a shorter, accurate one.
Six checks for a useful music caption
1
Describe the selected clip
Write about what is heard during this segment, not what happens elsewhere in the full track.
2
Name audible elements
Include instruments, voices, rhythm, tempo, or production features that can be supported.
3
Explain relationships
Say which element leads, supports, repeats, enters, or changes when that relationship is clear.
4
Use careful subjective language
Mood and energy can be useful, but they should be framed as descriptions rather than unquestionable facts.
5
Avoid hidden assumptions
Do not guess the artist, country, year, instrument model, or cultural tradition from sound alone.
6
Prefer accuracy over decoration
Extra adjectives are not helpful when they add no clear musical information.
Weak and useful captions
Swipe sideways to view the full comparison
Caption
Problem or strength
Training value
Track 42
Identifies a record but says nothing about the sound
Very weak for text-conditioned generation
Nice song
Subjective and almost empty of musical detail
Weak
Upbeat funk groove with slap bass and brass accents
Specific, compact, and based on audible properties
Strong when the details are correct
A legendary 1970s New York funk anthem by a famous band
Adds history, place, status, era, and identity that may not be audible or verified
Risky unless trusted metadata supports every claim
Fast instrumental groove led by slap bass, with syncopated drums and short brass responses
Describes elements and their relationship
Strong when supported by the clip
Unsupported labels
Some labels look plausible but are not supported by the available evidence.
A filename may say summer song, but the audio may not communicate a season. A catalogue may list jazz, while the selected ten-second segment contains only a spoken introduction. A caption writer may hear a plucked string and name a specific regional instrument without enough evidence.
Keep the source of each field clear:
Heard in audio for audible observations
Provided by source for catalogue or uploader metadata
Human interpretation for mood, genre, and other subjective labels
Model generated for automatic tags or captions
These sources should not be treated as equally certain.
Why genre labels conflict
Genre is not one universal measurement. It is a way of grouping music, and different communities may draw the boundaries differently.
One platform may use electronic as a broad category. Another may divide the same recordings into house, techno, trance, ambient, and many smaller styles. A track can also combine several genres, or sit between a genre, a scene, and a marketing label.
Research on genre-tag systems has shown that the same musical concept can appear under duplicate labels, translations, and different taxonomies. This is why combining metadata from several sources requires mapping rather than simple string matching.
In practice
Real annotation disagreement
MTG-Jamendo publishes an additional music-classification annotation set created with three annotators. Its clean version keeps examples where all three annotators agreed and removes answers marked unmatched or instrumental for the relevant taxonomy. The need for this filtering shows that even trained annotation work can produce disagreement.
How to handle inconsistent metadata
1
Preserve the original value
Keep the exact source label so it can be traced and reviewed later.
2
Record the source
Store whether the value came from an uploader, catalogue, annotator, model, or editor.
3
Map to a canonical value
Connect spelling variants, synonyms, translations, and related labels to a documented internal vocabulary.
4
Allow several labels
Do not force one genre when the track genuinely fits several categories.
5
Keep unknown values
Unknown is safer than inventing a confident label.
6
Track confidence and agreement
Save whether the label was verified and whether annotators agreed.
Captions generated from tags
Human-written music captions are expensive to collect. LP-MusicCaps explored a scalable alternative by asking a language model to turn existing tag sets into captions. The project produced millions of caption records from large music-tag datasets.
This can make language training data much larger, but the caption cannot know more than its inputs unless another source is used. If the tags say piano, slow, and sad, the generated sentence should not suddenly add a female singer, vinyl noise, or a film soundtrack.
Automatically generated captions should therefore keep links to their source tags, generation prompt, model version, and review status.
In practice
Why captions matter for generation
MusicGen is conditioned on textual descriptions or melodic features. During text-conditioned generation, the description acts as the user's control signal. A training dataset therefore needs language that consistently connects words such as instruments, tempo, style, and mood with matching audio.
captioned-music-record.json · A record that keeps original metadata, normalised tags, caption evidence, and provenance separate.
Try it: Inspect the segment evidence before reading the captions. Mark each musical claim as supported, uncertain, unsupported, or not audible in this segment. Then rate specificity, usefulness, and clarity. Finish by approving the strongest caption.
A useful caption describes audible musical properties with enough detail to guide a model. It should fit the selected segment, avoid unsupported claims, and keep subjective descriptions such as mood separate from verified facts.
Check your understanding
Ready for a quick check?
Test whether you can separate metadata, tags, and captions, then judge whether a music description is useful and supported.
Ready to continue?
Save this section to your account and continue from any device.