Machine Learning

The idea

What does it mean to learn?

Chapter 1 ended with a turn: instead of writing rules, collect examples and let the machine find the pattern. This chapter makes that precise. Machine learning means choosing a family of functions with adjustable parameters, then using data to pick the parameters that make the function fit — in a way that keeps working on examples it hasn't seen.

Everything depends on where the training signal comes from. With labelled examples, you learn a mapping from inputs to answers (supervised). Without labels, you look for structure (unsupervised). And with only raw data, you can invent labels from the data itself — hide a word and predict it — which is self-supervised learning, the way every LLM is pretrained.

For supervised learning the data is a table you'll recognize from data engineering: rows are examples, columns are features, one column is the label. If the label is a number it's regression; if it's a category, classification.

How it works

Your first models: lines and curves

Every model in this site — up to GPT — follows the same three-part recipe you met in Chapter 2: a model with parameters, a loss that scores it, and an optimizer that lowers the loss. The simplest instance is linear regression: predict a number as a weighted sum of the features, and choose the weights that minimize the squared error.

Swap the output for a probability — pass the weighted sum through a sigmoid — and swap squared error for cross-entropy, and you have logistic regression, the basic classifier. Train one below and watch the boundary settle; then try the XOR data and watch it fail.

The central problem

The central problem: generalization

Here is the most important idea in machine learning. A model can always fit its training data better by becoming more flexible — but fitting the training data isn't the goal. Predicting new data is. Move the slider and watch the two errors part ways.

At degree 11, the curve passes exactly through all twelve training points — zero training error — and is wildly wrong everywhere else. That is overfitting: memorizing noise instead of learning the pattern. At degree 0 the model is too simple to capture anything — underfitting. The job is to find the middle, and the only honest way to find it is to measure on data the model never trained on.

When you can't simplify the model, you can penalize complexity instead. Turn on regularization in the lab at degree 11 and the test error drops back near the best simple model.

Failure modes

How ML goes wrong

Two failures account for a large share of models that look great in evaluation and disappoint in production — and both are, at heart, data pipeline problems.

Leakage: the training data contains information that won't exist at prediction time — a feature computed from the future, a preprocessing step fitted on the test set, the same customer in both splits. The model learns the leak, not the task.

Distribution shift: the world the model is deployed into differs from the one it was trained on. A test-set score is a promise only about data like the test set.

Measurement

Measuring a classifier

Accuracy is the obvious metric and often the wrong one. When one class is rare — fraud, disease, spam — a model that always says "no" scores high accuracy while being useless. The fix is to count which kind of mistakes the model makes, and to choose where on the precision–recall trade-off you want to be.

The toolbox

The classical toolbox

You don't need to memorize algorithms — but you should recognize the main families and what each is good for. Each is a different answer to the same question: what shape of function should the model be allowed to learn?

The limit

From hand-crafted features to learned ones

Every model in this chapter learns only the last step: from features to answer. Someone still has to design the features — and for images, audio and text that was years of expert work per domain, the knowledge-acquisition bottleneck from Chapter 1, one level down. The XOR example in the logistic regression lab is this problem in miniature: a linear model fails until someone invents the right feature.

The next step is to learn the features too. That is what neural networks do.

Why it matters

Why it matters

Everything later is this chapter at scale. An LLM is a model with parameters, trained by gradient descent on a loss (cross-entropy) over data, and evaluated on held-out benchmarks. Overfitting, leakage (called contamination for LLMs), distribution shift and misleading metrics are exactly the failure modes you'll meet in Chapter 16.

For your day job, much of this is directly usable. On tabular data — the kind data engineers own — well-featured linear models and tree ensembles remain strong, fast and explainable. Knowing when that's enough is as valuable as knowing how a Transformer works.