Rotary Position Embedding (RoPE)

Position as an angle

Chapter 7's positional encoding added a position vector to each token. RoPE does something else. Su and colleagues encode absolute position with a rotation matrix applied in self-attention, which makes the attention score depend explicitly on the relative position of the two tokens Established.

Split a query's coordinates into pairs, and treat each pair as a point in a plane. At position $m$, rotate the pair by the angle $m\theta$. Do the same to the key at position $n$. Rotations preserve lengths, and the dot product of two rotated vectors depends only on the angle between them:

$$\langle R(m\theta)\,q,\; R(n\theta)\,k \rangle = \langle q,\; R\big((n-m)\theta\big)\,k \rangle .$$

A tiny example

Take $q = k = (1, 0)$ and $\theta = 30°$.

• Query at position 2, key at position 5: rotated by 60° and 150°. They are 90° apart: score $\cos 90° = 0$.
• Query at 10, key at 13: rotated by 300° and 390° (= 30°). Again 90° apart: score 0.
• Query at 4, key at 5: 30° apart: score $\cos 30° \approx 0.87$.

Same distance, same score, wherever the pair sits in the sequence.

Many clocks

A real head has many pairs, each turning at its own speed. RoFormer sets the frequencies to $\theta_i = 10000^{-2(i-1)/d}$ Established, the same geometric spread as the original sine waves: fast pairs distinguish neighbours, slow pairs distinguish distant tokens. Llama 3 uses RoPE with a base of 500,000 Established; a larger base makes the slow pairs turn more slowly, which helps when positions run into the tens of thousands.

Going past the trained length

A model trained on 4,096 positions has never seen the angles of position 20,000. Chen and colleagues found that extrapolating RoPE beyond the trained length can produce catastrophically high attention scores; instead they linearly scale down the position indices to fit the original window, and extended LLaMA models to 32,768 tokens with fine-tuning of up to 1,000 steps Established. Llama 3 supports contexts of up to 128K tokens Established.