The same performance, shown in three different forms
Compare waveform audio, spectrograms, and MIDI, and learn why each representation reveals a different part of music.
About 12 minutes
Guiding question
Which representation contains the recorded sound itself?
By the end, you’ll be able to
Explain what a waveform, spectrogram, and MIDI file each represent.
Compare the musical information that each representation keeps and leaves out.
Choose a useful representation for audio generation, sound analysis, or symbolic composition.
A piano phrase can be stored as audio, displayed as a colourful frequency map, or written as a list of note events. These may look like three versions of the same thing, but they do not contain the same information.
A waveform represents the recorded audio signal over time. A spectrogram is calculated from that signal and makes its changing frequency content easier to see. MIDI stores musical instructions such as which notes were played, when they started, and how strongly they were pressed.
This difference matters in AI music because the representation shapes the task. A model working with waveforms must learn the details of sound itself. A model working with MIDI can focus more directly on notes, timing, and musical structure.
Three representations at a glance
Swipe sideways to view the full comparison
Representation
What it stores or shows
What it is useful for
What it does not tell you directly
Waveform
The audio signal as changing sample values over time
Playback, editing, synthesis, vocals, timbre, and production detail
Which notes or instruments created the sound
Spectrogram
How frequency content changes over time
Finding harmonics, noise, percussion, pitch regions, and spectral patterns
A simple list of notes or a complete musical score
MIDI
Timed musical events such as note on, note off, velocity, controllers, tempo, and instrument instructions
Composition, arrangement, editing, transcription, and symbolic generation
The final recorded sound of a voice or instrument
1. Waveform: the recorded signal
A waveform display places time on the horizontal axis and amplitude on the vertical axis. Large peaks show moments where the recorded signal has a larger amplitude. Quiet sections sit closer to the centre line.
The waveform is connected to the actual audio samples stored in the file. This means it carries the full mixture of information in the recording, including the performance, instrument tone, voice, room sound, effects, background noise, and final production.
That makes waveforms rich, but also demanding. One second of audio can contain tens of thousands of samples per channel. A model that works directly at waveform level must handle very long sequences even for a short sound.
Waveform — A representation of how an audio signal changes over time, usually shown with time on the horizontal axis and amplitude on the vertical axis.
Example: A drum hit appears as a sharp peak because its amplitude changes quickly.
In practice
AI example: WaveNet
WaveNet is an early example of a model designed to generate raw audio waveforms. It predicted one audio sample at a time based on the samples that came before it. The original researchers showed that it could generate speech and short musical fragments. The model also shows why waveform generation is difficult: even a short clip contains a very large number of samples.
2. Spectrogram: a map of frequency over time
A waveform makes amplitude changes easy to see, but it does not clearly show which frequencies are present. A spectrogram adds that missing view.
A spectrogram is calculated by dividing the audio signal into short sections and analysing the frequency content of each section. The horizontal axis represents time, while the vertical axis represents frequency. Colour or brightness shows the strength of the frequency content at each point.
Low bass energy appears near the bottom. Higher sounds appear further up. A steady musical tone often creates horizontal bands, while a short drum hit spreads energy across a narrow moment in time.
A spectrogram is still based on audio. It is not a score and it does not automatically know that a bright band is a violin, a voice, or a synthesizer. It simply gives us a clearer view of how the signal's frequency content changes.
Spectrogram — A time and frequency representation that shows how the frequency content of a signal changes.
Example: A sustained piano note appears as several horizontal bands because the note contains a fundamental frequency and harmonics.
In practice
AI example: GANSynth
GANSynth generated musical instrument sounds in the spectral domain rather than predicting each waveform sample directly. The model worked with log magnitudes and instantaneous frequencies, then converted the generated representation back into audio. This is a useful example of how a model can learn from a frequency-based representation while still producing a sound that people can play and hear.
3. MIDI: instructions for a performance
MIDI takes a different approach. It does not store a recording of a piano, drum kit, or singer. It stores messages that tell a compatible instrument or sound engine what to do.
A MIDI performance can include:
A note-on event that starts a note
A note number that identifies the key or pitch
Velocity, which often reflects how strongly a key was pressed
A note-off event that ends the note
Program changes that select an instrument sound
Controller messages for actions such as sustain, modulation, expression, and pitch bend
Timing, tempo, track, and sequence information in a Standard MIDI File
The final sound is produced later by a synthesizer, sampler, digital piano, or software instrument. This is why the same MIDI file can sound like a concert piano on one system and a bright electronic keyboard on another.
MIDI — A technical standard for sending and storing musical performance instructions as timed events.
Example: A MIDI file can tell a synthesizer to play middle C with a certain velocity without containing a recording of a piano.
The piano roll
MIDI is often displayed as a piano roll. Time runs from left to right, pitch runs from low to high, and each rectangle represents a note. The position shows when the note begins, the length shows how long it lasts, and colour or brightness can be used to show velocity.
This view is easy to edit. A note can be moved, shortened, deleted, or copied without changing the rest of the performance. That makes symbolic data useful for studying melody, harmony, rhythm, and song structure.
In practice
AI example: Music Transformer
Music Transformer treated musical performances as long sequences of events. This allowed the model to focus on timing, notes, motifs, and repeated musical structure instead of learning the sound of a piano recording. The researchers used it to generate minute-long piano performances, continue musical ideas, and create accompaniments conditioned on melodies.
Which representation should you choose?
Swipe sideways to view the full comparison
Goal
Useful starting representation
Reason
Generate finished audio with vocals, instruments, and effects
Waveform or another audio representation
The model needs information about the sound itself
Inspect frequencies, harmonics, noise, or percussion
Spectrogram
Frequency patterns become easier to see across time
Generate melodies, chords, or arrangements
MIDI or another symbolic representation
Notes and timing are explicit and easy to edit
Turn a performance recording into editable notes
Waveform or spectrogram input, MIDI output
The task converts audio into symbolic events
Turn notes into a realistic performance
MIDI input, audio output
The task combines musical instructions with sound synthesis
In practice
One performance, two useful forms
The MAESTRO dataset contains paired piano audio and MIDI recordings aligned to within about 3 milliseconds. The audio captures the sound of the performance, while the MIDI records note events, key velocities, and pedal movements. Researchers can use the pair to study audio transcription, symbolic generation, and the conversion of generated notes back into realistic sound.
Interactive lesson
See One Performance in Three Forms
Try it: Move the visual cursor through each view and compare what becomes easy to see in each representation. In the MIDI view, change the displayed instrument and notice that the notes stay the same while the sound description changes.
A waveform shows the recorded signal over time. A spectrogram shows how frequency content changes over time. MIDI stores timed performance instructions such as notes and velocity, but it does not contain the finished recorded sound.
Check your understanding
Ready for a quick check?
Test whether you can tell the three music representations apart and choose the right one for a task.
Ready to continue?
Save this section to your account and continue from any device.