Supervised Fine-Tuning

Show what success looks like

Give a model a task and a good answer, then train it to predict that answer. A summary demonstrates what to keep; a code answer demonstrates a function and its explanation; a conversation demonstrates when to respond. This is ordinary supervised learning applied to a pretrained language model.

Instruction tuning is a common form of SFT: tasks are expressed as instructions. FLAN and T0 explored whether training on many such tasks transfers to instructions from held-out task families. That split matters: remembering examples is easier than following new kinds of request.

Same loss, different targets

Imagine the sequence 2 + 2 ? Four . END. In a response-only setup, the first four tokens provide context and the last three receive loss. Teacher forcing supplies the dataset prefix: when predicting the period, the model is given Four, even if it would have sampled a different word.

For an authored example, probabilities of 0.50, 0.25 and 0.80 for those three targets give losses of 0.693, 1.386 and 0.223 nats. Their mean is 0.768 nats. Changing a masked prompt probability leaves that mean unchanged.

$$\mathcal L_{\mathrm{SFT}}=-\frac{\sum_t m_t\log p_\theta(z_t\mid z_{<t})}{\sum_t m_t}$$

Here $m_t$ is 1 for a supervised target and 0 otherwise. A mask with no targets has no mean loss. This formula shows one sequence's mean; a training system must also choose how to aggregate across batches and lengths.

The mask removes a direct loss term, not the prompt's role in the computation. The response predictions still depend on prompt representations, so response loss can send gradients through them.

What changed, and what did not

SFT changes weights. In-context learning changes the supplied context without updating them. Both still use the causal next-token mechanism.

Post-training can change task performance as well as presentation. The useful question is empirical: which held-out abilities improved, which regressed, and under what evaluation? A narrow demonstration set can teach a format without robustly teaching the underlying task.

Try the distinction

In the lab, use Assistant only, move the prompt slider, then switch to All tokens. Explain why the first move leaves the loss unchanged and the second changes what the model is asked to learn. Finally clear the mask: no selected targets means no learning signal for this example.