Superposition and Polysemantic Neurons

Neurons that mean several things

If every neuron in a model detected one concept, you could read the model by listing what each neuron responds to. Real models disappoint. In the small Transformer studied in Towards Monosemanticity, a single neuron responds to a mixture of academic citations, English dialogue, HTTP requests and Korean text. Established Neurons like this are polysemantic. As early as 2013, Szegedy and colleagues found no distinction between individual high-level units and random linear combinations of them under the unit-analysis methods they tried, suggesting that the space, not the units, carries the meaning. Established

More features than dimensions

A model may need to track far more features than it has neurons: every name, topic, language, syntactic role. In $m$ dimensions you can fit only $m$ exactly perpendicular directions, but you can fit many more that are almost perpendicular. If two features are rarely active together, giving them overlapping directions costs little.

Elhage and colleagues demonstrated this with toy models: $h = Wx$ compresses $n$ sparse features into $m < n$ dimensions and $x' = \mathrm{ReLU}(W^\top W x + b)$ tries to reconstruct them. With dense features, the model learns an orthogonal basis for the most important features, like PCA, and drops the rest; as features become sparse, it represents more of them in superposition, first as antipodal pairs and then in geometric arrangements such as pentagons. Established

Why the ReLU matters

With overlapping directions, switching on feature 1 also nudges the outputs of its neighbours: interference. The paper finds that a linear model never uses superposition, while the ReLU output model does, using a negative bias so that small interference falls below zero and is removed. Established The model accepts a little error when two features happen to fire together, in exchange for representing many more features the rest of the time.

Tiny example (lab numbers)

In the lab, five features with importances 1, 0.7, 0.49, 0.34 and 0.24 must pass through two hidden numbers.

• Dense (every feature active on every input): after training, features 1 and 2 have length 1.0 and sit 90° apart; features 3–5 have length about 0 and their biases settle near 0.5, the average value of a feature. Two features, two dimensions.
• 30% density: four features, as two antipodal pairs (opposite directions), at 90° to each other. The least important feature is dropped.
• 3% density: all five, at roughly 72° apart, a pentagon, with biases near −0.25 to filter interference. (With decaying importances this happens on 6 of 10 random seeds; the other 4 keep four features. With equal importances, 10 of 10 form a pentagon.)

In real models

Anthropic's Towards Monosemanticity decomposed the 512-neuron MLP layer of a one-layer Transformer into thousands of features with sparse autoencoders, finding features far more interpretable than the neurons, consistent with superposition. Established The picture that results: a layer's activation is a sum of a few active feature directions out of a very large dictionary, and a neuron is just one coordinate of that sum. Interpretation Sparse autoencoders are the tool for recovering the dictionary.

Mini experiment

In the lab, train at 3% density and click a feature to switch it on alone. Which other outputs move, and why don't the ones on the far side of the pentagon? Then train at 100% density and repeat. Where did features 3–5 go?