Speech Recognition and Synthesis

The alignment problem

A 3-second clip at 10 ms per frame is 300 frames; its transcript "hello world" is 11 characters. Which frames are the "h"? A frame-by-frame classifier would need that answer for every training example.

Three generations

1. 1

   Pipelines

   For decades, recognisers combined an acoustic model (Gaussian mixtures over spectral features), a pronunciation dictionary and an n-gram language model, stitched together with hidden Markov models. Around 2012, four research groups reported that deep neural networks beat the Gaussian-mixture acoustic models Established.
2. 2

   CTC

   Connectionist temporal classification lets the network output a character or a special blank at every frame. To read the output, merge repeats and delete blanks: hh_e_ll_llo → hello (the blank between the two l-runs keeps the double l). Training sums the probability of every frame sequence that collapses to the true transcript, computed efficiently by dynamic programming. No alignment labels needed.
3. 3

   Encoder–decoder

   Treat recognition as sequence-to-sequence translation: an encoder reads spectrogram frames and a decoder writes text tokens, using attention to decide which frames to look at. Whisper trained such a Transformer on 680,000 hours of multilingual audio paired with transcripts from the internet, and transcribes, translates and identifies languages without task-specific fine-tuning Established.

The arc mirrors the rest of this site: hand-built components replaced by one learned model, then one bigger model trained on much more (and messier) data.

The reverse direction

Text-to-speech typically predicts a spectrogram-like representation from text, then a vocoder turns it into a waveform. WaveNet (2016) showed that generating raw audio sample by sample with dilated causal convolutions sounds far more natural than earlier synthesis Established. Many recent systems instead predict discrete audio tokens with a language-model-style Transformer (Chapter 15).