Preference Optimisation

Fine-tuning taught the model to imitate good examples. If the dataset contains a helpful burger answer, supervised fine-tuning makes that kind of answer more likely.

How do we train a language model when there is no single correct answer?

Many real prompts have multiple valid answers. One answer may be shorter, another more useful, another safer, another better formatted. Preference optimisation asks a new question: among possible answers, which one would people prefer?

Fine-tuning asks what a good answer looks like. Preference optimisation asks which answer is better. That contrast is the heart of this chapter.

Imagine Jessie is training at a burger restaurant. At first, she watches good employees answer customers. That is supervised fine-tuning.

Later, her manager shows two possible replies and asks which one should be used:

The manager chooses Response B. Jessie learns something more subtle than imitation: useful answers tend to be specific, constraint-aware, practical, safe, and neither too short nor too long.

Preference optimisation tries to teach that judgement. It is useful because comparison is often easier than creation. A person may struggle to write the perfect answer, but can still recognize which of two answers is better.

A preference dataset usually contains triples:

Here $x$ is the prompt, $y_w$ is the preferred response, and $y_l$ is the rejected response.

{
  "prompt": "I only have beef, buns, onion, and cheese. What can I make?",
  "chosen": "Make a cheeseburger. Shape the beef into a patty, cook it in a pan, melt the cheese on top, toast the bun, and add fried onion.",
  "rejected": "Make food with the ingredients."
}

The label does not say the chosen response is the only possible correct answer. It says that, for this prompt, this response is better than that one.

Close comparisons are especially valuable. If the rejected answer is nonsense, the model learns little. If both answers are plausible but one is safer, clearer, or more constraint-aware, the model learns judgement.

One way to learn from preferences is to train a reward model. A reward model takes a prompt and response and outputs a scalar score:

A higher reward should mean the response is more likely to be preferred. For a preferred response $y_w$ and rejected response $y_l$, we want:

Bradley-Terry Loss

A common differentiable model says the probability that humans prefer the winner is:

The difference matters because preferences are relative. The loss for one pair is:

If the reward model gives the winner score $5$ and the loser score $2$, the gap is $3$, $\sigma(3)\approx0.95$, and the loss is small. If the scores are reversed, the loss is large.

The reward model does not need perfect absolute scores. It needs to rank preferred responses above rejected ones.

RLHF stands for reinforcement learning from human feedback. The classic pipeline has three broad stages: pretrain a language model, fine-tune it on demonstrations, then use preference data to train a reward model and optimise the language model against that reward.

Let $\pi_\theta(y\mid x)$ be the language model policy and $r_\phi(x,y)$ be the reward model. A naive objective would be:

But optimizing only reward is dangerous. The model can exploit weaknesses in the reward model and drift away from useful behavior. So RLHF usually keeps the model close to a reference model, often the supervised fine-tuned model:

The reward term pulls toward preferred behavior. The reference penalty keeps the model from moving too far from a model we already trust. The coefficient $\beta$ controls that trade-off.

Reward Hacking

Reward hacking happens when the model gets high reward without satisfying the true goal. If a reward model overvalues confident wording, the assistant may become overconfident. If it overvalues long answers, the assistant may become bloated.

The measured objective is not always the real objective. This is why models need human evaluation after preference optimisation, not only reward-model scores.

DPO, or direct preference optimisation, learns from preference pairs without training a separate reward model and without running a full reinforcement learning loop.

It asks: can we directly update the language model so it assigns higher probability to preferred responses than rejected responses?

Let $\pi_\theta$ be the policy we are training and $\pi_{\text{ref}}$ be the frozen reference model. For one preference pair, DPO defines:

The first ratio asks how much more the new model favors the winner compared with the reference. The second ratio asks how much more it favors the loser. The difference asks whether the policy shifted more toward the winner than toward the loser.

DPO folds reward learning and policy optimization into a supervised-style objective. It is simpler than classic RLHF, but it still depends heavily on high-quality preference data.

Human preference labels are expensive, and some tasks can expose labelers to harmful content. Constitutional AI uses a written set of principles, a constitution, to guide critique, revision, and AI-generated feedback.

RLAIF means reinforcement learning from AI feedback. Another AI system compares or critiques responses according to principles or rubrics. This can scale feedback, but it does not remove the need for human judgement. Human judgement moves upstream into designing principles, audits, and evaluations.

Preference optimisation can teach many qualities, but it only optimises what the feedback process rewards.

These goals can conflict. A request for harmful instructions may require refusing the harmful part while still offering safe alternatives. A bad safety model refuses too much. A good one balances helpfulness, harmlessness, and honesty.

Preference optimisation changes model behavior. Retrieval gives the model external information. If the problem is missing current facts, use retrieval. If the problem is poor judgement or constraint-following, use fine-tuning or preference optimisation.

Reward Model

for batch in dataloader:
    prompt = batch["prompt"]
    chosen = batch["chosen"]
    rejected = batch["rejected"]

    reward_chosen = reward_model(prompt, chosen)
    reward_rejected = reward_model(prompt, rejected)

    difference = reward_chosen - reward_rejected
    loss = -log_sigmoid(difference)

    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

DPO

for batch in dataloader:
    prompt = batch["prompt"]
    chosen = batch["chosen"]
    rejected = batch["rejected"]

    chosen_logp = policy.log_prob(prompt, chosen)
    rejected_logp = policy.log_prob(prompt, rejected)

    with torch.no_grad():
        chosen_ref_logp = reference.log_prob(prompt, chosen)
        rejected_ref_logp = reference.log_prob(prompt, rejected)

    chosen_shift = chosen_logp - chosen_ref_logp
    rejected_shift = rejected_logp - rejected_ref_logp

    z = beta * (chosen_shift - rejected_shift)
    loss = -log_sigmoid(z)

    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

The policy model is trained. The reference model stays frozen. The objective rewards the policy for shifting probability toward the chosen response more than toward the rejected response.