Concept · Chapter 5: Vision, Speech & Reinforcement Learning
Images as Tensors
A digital image is a grid of pixels, each holding a few numbers, so a colour photo is a height × width × 3 tensor.
The problem
A network needs numbers, but an image's meaning lives in spatial patterns, not in any single pixel.
The solution
Store brightness per colour channel at every grid position, keeping the rows and columns so neighbouring pixels stay neighbours.
The consequence
Images are large inputs with strong local structure, which is exactly what convolutions are designed to exploit.
You should understand first
- Vectors
- Tensors and Shapes
- Images as Tensors
A grid of numbers
A grayscale image is a matrix: one brightness value per pixel, often stored as an integer from 0 (black) to 255 (white) and rescaled to 0–1 before training. A colour image stacks three such matrices, one per channel: red, green and blue. An orange pixel might be (255, 128, 0).
So a colour photo is a 3-D tensor of shape height × width × channels, and a training batch adds a fourth axis for the batch. Libraries disagree about the order: PyTorch usually stores channels first (batch × 3 × H × W), TensorFlow channels last.
Why this matters for learning
Even a small 224 × 224 colour image is 150,528 numbers. Connect each to 1,000 hidden units of a fully connected layer and you already need about 150 million weights for the first layer alone. Worse, flattening the grid into one long vector discards the fact that pixel (10, 10) sits next to pixel (10, 11). A fully connected layer must learn that relationship from scratch, separately at every location.
Images have two properties a model can exploit:
- Locality. Useful patterns such as edges, corners and textures live in small neighbourhoods.
- Translation. A cat's ear is a cat's ear in the top-left or the bottom-right of the photo.
Convolution builds both assumptions into the layer itself.
What to remember
- Grayscale: one number per pixel. Colour: three (red, green, blue).
- A 224 × 224 colour image is 224 × 224 × 3 = 150,528 numbers.
- Neighbouring pixels are strongly related; flattening the grid throws that layout away.