Lunar Boom Learning
3.2 · Autoregressive Music Models
Section 3.2 of 3.6
Building music one token at a time
Learn how autoregressive models repeatedly predict, select, and append tokens, why every choice changes the next context, and where sequential generation becomes difficult.
About 13 minutes
What happens after each token is generated?
Section 1 stopped after one prediction. An autoregressive model repeats that prediction until it has built a sequence.
The loop is simple:
The model does not normally sketch the complete song first. It moves forward one decision at a time.
Begin with a start token, text prompt, melody prompt, audio prompt, or existing sequence.
The model produces scores and probabilities for the possible next tokens.
A decoding rule uses greedy selection or sampling to choose one candidate.
The selected token becomes part of the generated sequence.
The model predicts again using the new sequence as context.
Generation ends at a stop token, requested duration, maximum token count, or another stopping rule.
Swipe sideways to view the full comparison
| Step | Context before prediction | Selected token | Context after selection |
|---|---|---|---|
| 1 | C4, E4, G4 | C5 | C4, E4, G4, C5 |
| 2 | C4, E4, G4, C5 | REST | C4, E4, G4, C5, REST |
| 3 | C4, E4, G4, C5, REST | G4 | C4, E4, G4, C5, REST, G4 |
| 4 | C4, E4, G4, C5, REST, G4 | E4 | C4, E4, G4, C5, REST, G4, E4 |
During training, the full sequence is available in the dataset. Without a restriction, a Transformer could inspect the token it is supposed to predict.
A causal mask blocks attention to future positions. When predicting token 12, the model can use tokens 1 through 11 but cannot use token 12 or anything after it.
This keeps the training task consistent with generation, where future tokens do not yet exist.
An autoregressive model represents the probability of a full sequence by multiplying its next-token probabilities:
P(x1, ..., xT | c) = P(x1 | c) × P(x2 | x1, c) × ... × P(xT | x1, ..., x(T-1), c)
Here, c represents optional conditioning such as a text description or melody.
Every prediction depends on the prefix before it. This is why token order matters and why changing one token can influence all later probabilities.
Swipe sideways to view the full comparison
| Mode | Initial context | Example |
|---|---|---|
| Unconditional generation | A start marker or minimal seed | Generate music without a text or melody prompt |
| Text-conditioned generation | A start marker plus a written description | Generate from 'warm lo-fi piano loop' |
| Melody-conditioned generation | A written prompt plus a melody representation | Arrange a hummed or instrumental melody |
| Audio continuation | An existing audio-token prefix | Continue a short piano recording |
| Symbolic continuation | Earlier note-event tokens | Continue a supplied piano motif |
Swipe sideways to view the full comparison
| Model type | Predicted unit | Sequence consequence | Example |
|---|---|---|---|
| Raw waveform model | One sample value | Tens of thousands of sequential positions per second | WaveNet |
| Symbolic music model | Note, time, velocity, or control event | A shorter sequence but no direct production audio | Music Transformer |
| Codec language model | Compressed audio token | Shorter than waveform samples while retaining reconstructable sound | MusicGen |
| Multi-level audio model | Tokens at several temporal or acoustic scales | Different autoregressive stages handle structure and detail | AudioLM and Jukebox |
Examples in practice
WaveNet generated raw waveforms one sample at a time. Every sample distribution depended on all previous samples. This produced realistic local audio, but generation was sequential at a very fine time scale because one second of audio contains tens of thousands of samples.
Music Transformer generated symbolic piano events autoregressively. Its relative-attention mechanism was designed to handle long event sequences and helped the model repeat and develop motifs over minute-long performances.
MusicGen uses one Transformer language model to predict several streams of EnCodec tokens. Its interleaving patterns arrange tokens from several codebooks into a sequence that can be generated autoregressively. Text and melody conditions steer the probabilities throughout generation.
Chapter 1 explained that residual vector quantisation can represent audio with several codebooks. A model then receives several token streams for the same part of the audio.
MusicGen avoids using a separate model for every stream. It uses one language model and an interleaving pattern that shifts or arranges the codebook tokens into a prediction sequence.
The ordering matters because it decides which codebook tokens are already available when another token is predicted. This is a design choice inside the autoregressive model, not a change to the neural codec.
A model cannot necessarily keep every earlier token available forever. The context window determines how much of the sequence can be used for the current prediction.
If the complete generated prefix fits inside the window, the model can use all of it. Once the sequence becomes longer, the system may need to discard old tokens, use a sliding window, compress earlier information, or apply another long-sequence strategy.
When old material leaves the usable context, the model may struggle to return to an opening motif or follow a plan introduced much earlier.
Autoregressive prediction is naturally suited to local continuity. The next note, drum hit, or audio token is strongly related to what happened just before it.
A complete song also needs larger relationships: sections, repetitions, tension, release, transitions, and returns. These relationships may span hundreds or thousands of tokens.
Music Transformer improved access to long-range symbolic relationships with relative attention. AudioLM and Jukebox used several token scales or stages so coarse structure and fine acoustic detail did not have to be handled in exactly the same way.
Suppose two generations begin from the same prompt. At step 20, one selects a held chord and the other selects a drum fill.
Those tokens change the context. The model may now assign different probabilities to the harmony, rhythm, instrumentation, and structure that follow. After another hundred steps, the two pieces can sound unrelated.
This is not automatically an error. It is one source of creative variation. It becomes a problem when an early choice moves the sequence away from the prompt, damages the timing, creates an unstable texture, or begins a pattern the model cannot resolve.
During standard next-token training, the model usually predicts from the correct earlier tokens stored in the dataset. This guided setup is commonly called teacher forcing.
During generation, the model receives its own selected tokens instead. If it enters a context that was rare or absent during training, its later predictions may become less reliable.
This difference between training on correct prefixes and generating from model-produced prefixes is often called exposure bias. The Scheduled Sampling paper was motivated by the observation that errors can accumulate when a sequence model must continue from its own previous predictions.
Token 100 depends on token 99, which depends on token 98. The model cannot finalise all three at the same time because the later inputs do not exist yet.
Training can process many positions in parallel because the complete correct sequence is already available and a causal mask prevents information leakage. Generation remains sequential because each selected token must be produced before the next step begins.
This helps explain why large autoregressive audio models can be expensive or slow to sample, especially when the representation uses many tokens per second.
Swipe sideways to view the full comparison
| Area | Strength | Limitation |
|---|---|---|
| Local continuity | Each prediction responds directly to the recent sequence | Strong local transitions do not guarantee a full-song plan |
| Flexible conditioning | Text, melody, artist, genre, or audio prompts can steer probabilities | Weak training labels can reduce control |
| Probabilistic variation | Sampling can produce several continuations from one prompt | Low-probability choices can create unstable paths |
| Likelihood training | Every token position supplies a clear prediction target | Training on clean prefixes differs from generation on model prefixes |
| Audio fidelity | Codec tokens can preserve detailed sound | More token streams increase generation work |
| Long sequences | Attention and compression can extend usable context | Context limits and serial decoding remain difficult |
Examples in practice
Jukebox compressed music into several levels of discrete codes and modelled those codes with autoregressive Transformers. A top-level prior handled a highly compressed sequence, while upsampling models added finer detail. The system preserved autoregressive generation but divided the problem across several scales.
AudioLM combined semantic tokens for long-term structure with neural codec tokens for acoustic quality. Its stages remained autoregressive, but different token types focused on different parts of the problem. Section 3 explains why this hierarchy can help with long audio.
Interactive lesson
Try it: Start with the same prompt and generate one token at a time. Inspect the distribution before selecting each token. At the marked branch point, create two versions by selecting different plausible tokens. Continue both branches and compare their later probabilities, structure, and visual token trajectories.
An autoregressive model predicts one token, selects it, appends it to the context, and repeats. Two different early tokens create different prefixes and can lead to increasingly different continuations. A limited context window may eventually remove older material from direct use.
A single autoregressive stream may need to handle both the structure of a song and tiny acoustic details. Those goals operate at very different time scales.
The next section, Hierarchical Generation, examines models such as AudioLM and Jukebox that divide the problem into levels. One stage can plan a compressed structure, while later stages turn that plan into detailed sound.
Check your understanding
Test whether you can follow the autoregressive loop and explain context limits, branching, exposure bias, and serial generation.
Save this section to your account and continue from any device.