Compute and Infrastructure

A model architecture can be mathematically valid and still be impossible to train with the available hardware.

Infrastructure determines which model sizes, sequence lengths, batch sizes, datasets, and experiments can be run in practice. It also determines whether a failed job can resume, whether two runs can be compared, and whether the result can be reproduced later.

The central question is therefore not simply Which GPU is fastest? It is:

Which combination of hardware, memory strategy, storage, software, and experiment controls can run this specific training plan reliably?

What occupies training memory?

Loading model weights is only the beginning. A conventional training step may require memory for:

1. Model parameters
2. Parameter gradients
3. Optimiser states
4. Forward-pass activations saved for backpropagation
5. Input batches, targets, masks, and conditions
6. Temporary operation buffers
7. Mixed-precision scaling or master values
8. Distributed communication buffers
9. Framework allocator reserves

The largest component depends on the model, optimiser, sequence length, batch size, precision, and implementation.

Allocated and reserved memory

PyTorch uses a caching allocator for CUDA memory. Memory released by tensors may remain reserved by PyTorch so it can be reused quickly.

As a result, a system monitor can show memory as occupied even when that memory is not currently assigned to live tensors.

Useful measurements include:

1. Memory allocated by tensors
2. Memory reserved by the caching allocator
3. Peak allocated memory
4. Peak reserved memory

This distinction is important when debugging apparent memory leaks.

Audio duration becomes sequence length

A music model does not usually receive thirty seconds as an abstract unit. The representation converts those seconds into samples, frames, latent positions, or tokens.

Approximate sequence length depends on:

1. Duration
2. Sample or frame rate
3. Codec token rate
4. Number of codebooks or token streams
5. Channel handling
6. Interleaving pattern
7. Padding and special tokens

Doubling duration usually increases the amount of represented data, but the exact memory and compute effect depends on the architecture.

Why long sequences are expensive

Longer sequences increase the number of token representations and layer activations that must be processed.

For conventional dense self-attention, the attention relationship between sequence positions forms a square matrix. Materialising this matrix creates memory and computational costs that grow strongly as sequence length increases.

Efficient attention implementations such as FlashAttention reduce memory traffic and avoid storing some large intermediate matrices. They improve feasibility and throughput, but they do not make long-context training free.

Batch size and microbatch size

A larger batch processes more examples together. This commonly increases activation and input memory because more examples pass through the network at once.

The microbatch size is the number of examples processed in one forward and backward pass on one worker. The effective batch size can be larger through gradient accumulation and multiple workers.

Reducing the microbatch is often the first response to an out-of-memory error, but very small microbatches can reduce hardware utilisation or change optimisation behaviour.

Gradient accumulation

Gradient accumulation divides a large effective batch into several smaller microbatches.

Each microbatch runs a forward and backward pass. Gradients are accumulated, and the optimiser updates the parameters only after the selected number of microbatches.

This lowers the activation memory required at one moment compared with processing the full effective batch together. It also adds more sequential computation and requires correct loss scaling and gradient clearing.

Numerical precision

Numerical precision determines how tensor values are represented.

Common training formats include:

1. FP32
2. FP16
3. BF16
4. Mixed-precision combinations

Lower-precision operations can reduce tensor memory and improve throughput on compatible hardware. Some calculations remain in higher precision for numerical stability. FP16 training may use gradient scaling to reduce underflow risk.

Total memory does not necessarily fall in direct proportion to the bit width because optimiser states, master values, temporary tensors, and unsupported operations may remain in higher precision.

Activation checkpointing

Ordinary backpropagation keeps many forward activations in memory. Activation checkpointing saves only selected tensors and recomputes omitted forward operations during the backward pass.

This trades additional computation for lower activation memory. It is especially useful when activations, rather than parameters or optimiser state, are the main bottleneck.

This technique is unrelated to saving a model checkpoint to disk.

The GPU can be fast while training remains slow

Music data may require file reading, decompression, resampling, channel conversion, tokenisation, caption parsing, and random cropping before it reaches the model.

If this pipeline is slower than the GPU, the accelerator waits between batches.

Potential responses include:

1. Additional data-loader workers
2. Persistent workers
3. Pinned host memory where appropriate
4. Prefetching
5. Cached or precomputed codec tokens
6. Faster local dataset storage
7. Sharding large datasets
8. Moving expensive preprocessing outside the training loop

Each change should be measured because excessive workers or caching can create new CPU, memory, or storage bottlenecks.

From one GPU to distributed training

A single GPU is usually the simplest environment for debugging and small experiments. Distributed training becomes useful when:

1. The model or training state does not fit on one device
2. A larger global batch is required
3. One-device throughput makes the experiment impractically slow
4. Evaluation or generation needs to be parallelised

Adding GPUs introduces communication, synchronisation, process management, failure handling, and distributed checkpointing. More devices do not guarantee proportional speedup.

Saving training progress

A checkpoint is a persistent record created at a particular training state.

Different checkpoints serve different purposes:

1. Generation or inference
2. Comparing earlier and later models
3. Selecting the best validation checkpoint
4. Resuming an interrupted training run
5. Preserving milestones for an audit
6. Rolling back after a software or data error

A weights-only checkpoint is smaller and simpler, but it generally does not contain everything needed to resume the exact optimiser trajectory.

How often should checkpoints be saved?

Frequent checkpoints reduce the amount of work lost after interruption. They also increase storage use and can pause training while large states are written.

The appropriate interval depends on:

1. Cost per training step
2. Failure or interruption risk
3. Checkpoint size
4. Storage bandwidth
5. Validation frequency
6. Whether asynchronous checkpointing is available
7. How many historical states must be retained

Test checkpoint loading before depending on the recovery plan.

Storage can become a larger problem than the model

A music project may need to store:

1. Original audio and MIDI
2. Cleaned and resampled versions
3. Captions and metadata
4. Precomputed codec tokens or embeddings
5. Dataset manifests and hashes
6. Model checkpoints
7. Optimiser states
8. Generated evaluation samples
9. Similarity indexes
10. Logs, plots, and profiler traces

Keeping every artifact from every run can consume substantial space. Deleting everything except the final model removes evidence needed for debugging and reproducibility.

Local, cloud, and hybrid workflows

A local workstation offers direct access to fixed hardware and data. It is well suited to pipeline development, small models, short runs, and repeated debugging.

Cloud or cluster infrastructure provides access to hardware that may not be available locally and can support larger or temporary experiments. It also introduces remote storage, job scheduling, environment setup, data transfer, access control, and interruption handling.

A hybrid workflow develops and validates the smallest version locally, then runs a versioned configuration on larger remote infrastructure.

Experiment tracking

A model result is difficult to interpret without knowing how it was produced.

A useful run record includes:

1. Run identifier
2. Parent or comparison run
3. Code commit
4. Model configuration
5. Dataset and split versions
6. Tokenizer or codec version
7. Optimiser and scheduler
8. Batch and accumulation settings
9. Precision and distributed strategy
10. Hardware and worker count
11. Seeds
12. Training and validation metrics
13. Gradient and memory diagnostics
14. Checkpoint paths
15. Generated samples
16. Notes about failures or manual changes

The objective is not to collect every number. It is to preserve the information needed to explain and repeat the result.

Real-world example: AudioCraft training infrastructure

Meta's AudioCraft training pipelines are built on PyTorch, use Flashy for training-pipeline design, and use Dora as an experiment manager.

The framework supports named configurations, experiment grids, team-specific environment settings, cluster configuration, and model-specific solvers. This separates experiment definitions from the physical machine or cluster used to run them.

For MusicGen, AudioCraft combines the compression model, language model, conditions, optimiser, metrics, validation, and generation stages within the training system.

Reproducibility has several levels

Exact bit-for-bit repetition is not always available across PyTorch releases, devices, platforms, kernels, and distributed schedules.

A project can still aim for:

1. Configuration reproducibility: the same settings can be reconstructed.
2. Data reproducibility: the same dataset and splits can be identified.
3. Procedural reproducibility: the same pipeline can be executed.
4. Statistical reproducibility: repeated runs produce comparable distributions of results.
5. Artifact reproducibility: the selected checkpoint and outputs can be recovered.

Record seeds, but do not treat one seed as a complete reproducibility system.

When the system still fails

A model that fits in memory can still produce NaN losses, silent data errors, stalled workers, broken checkpoints, invalid audio, or gradients that never reach part of the network.

The next section, Debugging Model Training, turns logs, gradients, memory traces, generated samples, and controlled tests into a systematic debugging process.