Concept · Chapter 5: Vision, Speech & Reinforcement Learning
Audio and Spectrograms
A spectrogram turns a waveform into a time × frequency picture by measuring the frequency content of short, overlapping windows of sound.
The problem
Raw audio is a very long one-dimensional list of pressure samples, and the patterns that matter (pitch, vowels, consonants) are about frequencies changing over time.
The solution
Slide a short window along the signal, take the Fourier transform of each window, and stack the magnitudes as columns.
The consequence
Speech becomes a 2-D input that CNNs and Transformers can process like an image or a sequence of frames.
You should understand first
- Vectors
- Audio and Spectrograms
From air pressure to numbers
A microphone measures air pressure many times per second. At 16 kHz, one second of speech is 16,000 numbers; a ten-second clip is 160,000. Plotted, it is the familiar wiggly waveform. Loud versus quiet is easy to see; which vowel was spoken is not.
Intuition: a piano roll for any sound
The Fourier transform answers "how much of each frequency is in this signal?" Applied to a whole clip it loses when things happened. So apply it to short slices instead: a window of about 25 ms, moved along in steps of about 10 ms. Each window yields one column of frequency strengths; side by side, the columns form a spectrogram: time left to right, pitch bottom to top, brightness for strength. A held note becomes a horizontal line, a rising whistle a diagonal, a click a vertical stripe, and a voiced vowel a stack of harmonics whose loud bands (formants) identify the vowel.
The trade-off you cannot escape
A window of N samples at sample rate separates frequencies about Hz apart, but smears everything that happens within its seconds. At 8 kHz, a 64-sample window lasts 8 ms and resolves 125 Hz; a 512-sample window lasts 64 ms and resolves about 16 Hz. Make one sharper and the other blurs: it is the same uncertainty trade-off that appears throughout signal processing.
Try it · toy model
Slice a sound into short windows, measure each window's frequencies, and discover why no window length is sharp in both time and frequency.
What speech models actually use
Two tweaks on the plain spectrogram: frequencies are grouped into mel bands, spaced roughly linearly at low frequencies and logarithmically at high ones (closer to how pitch is perceived), and magnitudes are log-compressed. Whisper, for example, resamples audio to 16 kHz and computes an 80-channel log mel spectrogram on 25 ms windows with a 10 ms stride Established. Some models skip the spectrogram and learn directly from raw samples with 1-D convolutions; WaveNet (2016) generated raw waveforms one sample at a time Established.
What to remember
- Sample rate: 16,000 samples per second is common for speech; one second is 16,000 numbers.
- Each spectrogram column is the frequency content of one short window (about 25 ms, stepped every 10 ms).
- Short windows are sharp in time but blurry in frequency; long windows the reverse. No window is sharp in both.
- Speech models usually use log mel spectrograms: frequency bands spaced like human pitch perception, on a log (decibel-like) scale.
Key papers
WaveNet: A Generative Model for Raw Audio
Aaron van den Oord, Sander Dieleman et al. · 2016
Generated audio one sample at a time with a stack of dilated causal convolutions, and produced much more natural-sounding synthetic speech.
Watch
3Blue1Brown
But what is the Fourier Transform? A visual introduction.
Builds the intuition behind splitting a sound into its frequencies, the operation every spectrogram column performs.