Robustness and Prompt Sensitivity

Changes that should not matter

A person who can answer "What is the capital of France?" can also answer "capital of france??". Models usually can too, but not always, and the failures are often surprising. Three kinds of change are worth separating:

1. Harmless variation: paraphrases, typos, formatting, order of options.
2. Natural shift: the inputs in use differ from the test set (distribution shift): new products, new slang, another language.
3. Adversarial inputs: someone searches for an input that makes the model fail.

Formatting is a variable

Sclar and colleagues sampled many semantically equivalent prompt formats and found accuracy differences of up to 76 points for LLaMA-2-13B, with sensitivity persisting in larger models, with more in-context examples and after instruction tuning. Established HELM made robustness, measured under perturbations of the inputs, one of its seven standard metrics. Established If a result holds only in one format, it is a result about that format.

Adversarial examples

Szegedy and colleagues found that a hardly perceptible perturbation, chosen to maximize a network's error, makes it misclassify an image, and that the same perturbation often fools other networks trained on different data. Established Goodfellow and colleagues attributed this to models behaving too linearly in high dimensions and introduced the fast gradient sign method. Established

Tiny example. A linear score $w \cdot x$ over 1,000 inputs. Change each input by only $\varepsilon = 0.01$ in the direction of the sign of its weight. The score moves by $\varepsilon \sum_i |w_i|$; if the weights average 0.5 in size, that is $0.01 \times 1{,}000 \times 0.5 = 5$, enough to flip a confident decision, though no single input changed noticeably. More dimensions, more room for an attacker.

For language models

Text is discrete, so attacks search over tokens rather than nudging pixels, but the logic is the same: use the model's own gradients or feedback to find inputs it handles badly. Zou and colleagues optimized a 20-token suffix that made safety-trained models comply with harmful requests, and it transferred to other models. Established That turns robustness into a security question, which jailbreaks take up.

Measuring robustness

• Run each test item in several formats and paraphrases; report the mean and the spread.
• Keep a slice of the test set from a different time or source to estimate shift.
• For security-relevant uses, report attack success rates under a stated attacker, not average-case accuracy.

Mini experiment

Take five questions your team asks an assistant. Write each three ways: formal, terse with typos, and with the options in a different order. Run all fifteen. How many answers changed? Which version would you have put in a benchmark?