From Base Model to Assistant

The problem

The checkpoint is not the assistant

Chapter 9 ended with a checkpoint: a large collection of weights learned from trillions of tokens. It can continue code, stories, arguments and conversations. What it has not necessarily learned is that, when you ask a question, it should stop being a document and start being your assistant.

Give it a quiz question and one plausible continuation is another quiz question. Give it a carefully framed example conversation and it may answer well. The capability and the reliability of the behavior are different questions. A base model is not incapable of answering; its pretraining objective simply does not specify the conversational contract you have in mind.

Post-training works on that contract. The generation mechanism remains familiar: predict a token, append it, predict the next. The training signal changes what counts as a good continuation.

The distinction has practical consequences. In InstructGPT's evaluation, people preferred outputs from a 1.3B-parameter tuned model to those from the 175B GPT-3 base model Established. That is a result on the paper's prompt distribution, not a claim that the smaller model acquired every ability of the larger one. The users were judging answers, not parameter counts.

Demonstrations

Show the answer

Suppose the task is to summarize a short transport report in two sentences, preserve its uncertainty and add no facts. A demonstration contains the source paragraph, that instruction and a suitable summary. Thousands of such examples can teach the model that this is the kind of continuation to produce.

This is supervised fine-tuning, or SFT. It is the same forward pass, loss, backward pass and weight update from earlier chapters, applied to a different collection of examples. Instruction tuning is the version in which those examples describe tasks as instructions. FLAN and T0 study how training on many tasks helps with instructions from held-out task families.

For a data engineer, the important change is the unit of data. A cleaned document was enough for pretraining. Now a record needs a task, a response, role boundaries and a reason to trust that response. Coverage matters: if every demonstration is verbose and certain, brevity and appropriate uncertainty will be underrepresented.

The InstructGPT paper's first version reports 12,725 SFT training prompts, 33,207 reward-model training prompts and 31,144 PPO training prompts Established. Those are prompts, not response pairs or all the tokens consumed. The three datasets supply different signals; “more training data” alone does not describe the recipe.

The objective

What the loss sees

Consider an invented training sequence: 2 + 2 ? Four . END. Prompt tokens remain available as context, while a loss mask can select only the three assistant targets. With probabilities 0.50, 0.25 and 0.80, their negative log probabilities average to about 0.768 nats.

Masking the prompt does not make it invisible. The answer still depends on it. It means the training objective does not directly reward predicting the user's words. Some recipes supervise all tokens; some supervise assistant turns or only the final completion. The choice must agree with the intended task.

Start with Assistant only and move the prompt slider. Nothing happens to the loss. Now select All tokens: the same probability affects the objective. The model did not change; the definition of what it is being asked to learn did.

During training, the prefix comes from the demonstration. When predicting the period, the model is given Four, even if it would have guessed something else. This is teacher forcing. At inference, it conditions on its own generated tokens instead. A low demonstration loss therefore does not guarantee that a long sampled conversation will stay on track.

The interface

Conversation has a format

The application holds messages with roles: system, user, assistant. A causal language model receives a token sequence. A chat template connects the two by choosing role markers, turn terminators and the prefix at which the assistant should continue.

The tokens are model-specific. A perfectly good demonstration can be damaged by an extra beginning token, a missing terminator or the wrong assistant marker. Inspect one serialized conversation and its mask before processing a million of them. This is a schema check with behavioral consequences.

Role markers are learned conventions, not a mathematical firewall. They help establish who is speaking; they do not, by themselves, solve prompt injection or application security. Those questions return in the systems chapters.

Comparisons

Which answer is better?

A demonstration gives one answer to imitate. It does not rank the many other answers a model might generate. Comparing candidates can be easier than writing an ideal response from scratch.

Here is our invented source: a town tested a bus route for four weeks; ridership rose; the trial was too short to tell whether the increase would last; no launch date was announced. One candidate preserves those facts in two sentences. Another announces a permanent launch date. A third is faithful but uses five sentences. A fourth refuses unnecessarily.

Which should win? “Sounds confident” and “preserves the source” do not choose the same answer. A preference label only becomes meaningful alongside the prompt and judging rubric.

Four ranked responses yield six pair comparisons, but not six independent prompts. Keep all variants of a prompt on the same side of the train/test split. Record ties and disagreement. “Chosen” in a dataset means preferred under a judgment, not guaranteed true.

A learned judge

Learning a judge

To reuse those judgments, train a reward model. It reads the prompt and response and returns a scalar. The training loss asks the chosen response to score higher than the rejected one. A common conversion from scores to a preference probability is

$$P(\text{chosen wins})=\sigma\big(r_{\mathrm{chosen}}-r_{\mathrm{rejected}}\big).$$

Equal scores give 50–50. A difference of one gives about 73.1%. Only the difference matters: scores 100 and 99 express the same pair probability as 1 and 0. The scale is not a meter of truth.

Label a pair, then fit the judge. For a sharper demonstration, open the two rubric presets. Faithfulness first favors the grounded summary. Confidence first rewards the invented conclusion. The same fitting procedure succeeds at learning either set of labels. This tiny model fits four free scores; a real reward model must also generalize to unseen language.

Optimization

Reward with restraint

Now let the assistant generate its own responses. The reward model scores them; reinforcement learning changes the policy to favor higher-scoring responses. In the InstructGPT recipe, PPO performs those updates after SFT. This is a common form of reinforcement learning from human feedback, RLHF.

There is a catch. A model actively searching for high reward can discover mistakes that a passive validation set did not expose. One restraint is to penalize departure from a frozen reference, commonly the SFT model:

$$\text{maximize expected reward}\; -\;\beta\,D_{\mathrm{KL}}(\pi_\theta\;\|\;\pi_{\mathrm{ref}}).$$

The KL term says how much the distribution changed. It cannot say whether a claim is true. Our next lab makes the difference measurable: its flawed judge rewards the confident false response most highly. At β=1, optimization raises that response's probability from 20% to about 53%.

Lower the restraint. Reward rises while correctness falls. Then penalize the unsupported claim: changing the judge changes what optimization achieves. This is an exact three-action calculation, not a PPO simulation or an empirical LLM result.

+Deep diveWhy PPO has another old modelShould know

PPO compares a token's new probability with its probability under the policy that collected the rollout. Its clipped objective limits the incentive for large updates to that sampled policy. This old rollout policy is distinct from the reference policy used for the KL penalty. The rollout policy refreshes; the reference can remain fixed. Clipping is not a guarantee of a maximum policy distance or safe behavior.

A shorter route

The DPO shortcut

The reward-model-plus-RL route is not the only way to use preferences. Direct Preference Optimization, DPO, fits the language model directly to chosen/rejected pairs, using a fixed reference to measure how each response's probability changes.

The arithmetic fits on a napkin. The reference log probabilities are −4 and −3. The policy's are −3.5 and −3.2. The chosen answer gained 0.5 log units; the rejected answer lost 0.2. The relative difference is 0.7. Multiply by β=0.5, apply a sigmoid and take negative log: the pair loss is 0.533, down from 0.693 at the reference.

Try Both probabilities fall. DPO can improve the comparison while making both answers absolutely less likely, because the rejected answer loses more relative to its reference. It does not independently promise to raise every chosen answer's probability.

Reward model + PPODPO

1. 01

   Show an answer

   Demonstrations train an SFT policy. Keep a frozen reference copy.

2. 02

   Compare answers

   Judges rank candidates. A reward model learns to predict those comparisons.

3. 03

   Sample and improve

   The policy generates responses. PPO uses reward and a penalty for drifting from the reference.

These are common teaching pipelines, not mandatory stages for every assistant. DPO-based systems can still use a reward model elsewhere, for example to select demonstrations.

Vanilla DPO avoids a separately trained reward model and online sampling during preference fitting. A whole training pipeline can still use judges elsewhere. Llama 3's report describes reward-based candidate selection alongside SFT and DPO. Do not confuse one objective with the entire development process.

The source of judgment

Whose feedback?

Human comparisons take time. Models can help write demonstrations, critique drafts or compare responses. The important question is which signal they supply and how it is checked.

Constitutional AI uses explicit principles to guide critique and revision, and AI-generated preferences for a feedback stage. Humans still choose the principles and judging setup. AI feedback shifts annotation work; it does not eliminate human choices or automatically create reliable ground truth.

Another operation is to generate several candidates, select good ones and train on the selected responses. This rejection sampling supplies an SFT dataset; it is different from optimizing a policy on sampled rewards. It also explains why inference efficiency matters while building a model: the training pipeline may generate and judge many answers before a user ever sees one.

“Pretraining gives knowledge; post-training only changes style” is a useful first approximation that becomes misleading when treated as a law Interpretation. Reported recipes such as Tulu 3 target multiple skills and combine SFT, DPO and verifiable rewards. Chapter 14 takes up reasoning-oriented reinforcement learning; here, measure the task changes instead of assuming they must be cosmetic.

Behavioral boundaries

Safety is more than refusal

A system that always says no would pass a test consisting only of requests it should decline. It would also fail as an assistant. Safety tuning needs examples of appropriate responses and appropriate refusals, including benign requests that resemble problematic ones.

Helpfulness, truthfulness and harmlessness are related but separate aims. A polite hallucination, an accurate but inappropriate response and an unnecessary refusal fail in different ways. Training examples and preferences can shape these behaviors; they do not replace the wider system checks covered in Chapter 16.

The exam

Test the assistant

After each stage, ask what actually improved. Keep a development set for choosing the recipe and an unseen test set for the eventual claim. Use prompt-level splits, blind model identity in comparisons, randomize answer order and report uncertainty rather than just a win percentage.

Reward is useful training feedback, but it is not the exam. The reward-overoptimization paper studies divergence from a synthetic gold reward; work on sycophancy investigates feedback that favors agreement with a user's belief over a more accurate response. Neither finding justifies a universal claim that preference training always fails. They tell us what to test.

For an engineer, this chapter is a reminder that schemas, labels and evaluation splits are part of model behavior. For a researcher, it exposes a coupled learning problem: the policy changes, the judge is imperfect, and the metric can become a target. More optimization is only helpful when the signal continues to represent what you want.

What comes next

The assistant still has to run

The output is another checkpoint. It still predicts tokens, but demonstrations and feedback have shaped which continuations it favors. Deploying it creates a different set of problems: growing context memory, serial token generation, many simultaneous requests and the cost of adapting large weights. Chapter 11 goes inside the engineering that makes those systems practical.