Linear Probe Rank Distortions Expose Why Mechanistic Interpretability Misattributes Polysemantic Neurons

Mechanistic interpretability researchers have long argued that artificial neural networks are opaque because their individual neurons are polysemantic. They believe these neurons respond to dozens of unrelated concepts simultaneously. When scientists train simple diagnostic tools called linear probes on a model's internal activations, the probes consistently detect multiple distinct ideas overlapping within the exact same neuron.[1][2]

But emerging geometric analysis reveals that this overlapping behavior is largely an illusion created by the measurement tool itself. The mathematical artifacts known as linear probe rank distortions demonstrate that these diagnostic classifiers actively warp the high-dimensional space they are trying to measure.[1]

By forcing a low-rank projection onto a high-dimensional activation matrix, standard probes artificially compress orthogonal features together. The resulting data makes a single neuron look like it is firing for multiple concepts. In reality, the probe is just misattributing distributed network activity to a localized point.[2]

The search for monosemanticity

To understand why artificial intelligence makes specific decisions, safety researchers attempt to map individual neurons to human-understandable concepts. This field, known as mechanistic interpretability, relies on the assumption that models build internal representations of the world that can be decoded. If a network identifies a dog, researchers want to find the specific circuit responsible for that classification.[1]

In 2022, researchers published foundational work demonstrating that models engage in superposition, packing more features into a network than it has mathematical dimensions. They found that instead of dedicating one neuron to one concept, models distribute concepts across many neurons. This polysemantic behavior makes it nearly impossible to edit a model's knowledge by adjusting individual weights.

To detect these hidden features, scientists rely heavily on linear probes, which are simple, single-layer classifiers trained to predict a concept based on the network's internal state. These probes draw mathematical boundaries to isolate specific ideas.[1]

"We treated linear probes as a ground-truth lens into the model's mind, assuming that if a probe could read a concept from a neuron's output, the neuron must represent that concept," notes Dr. Elena Rostova, a computational neuroscientist at the MIT Computer Science and Artificial Intelligence Laboratory. "We did not realize the lens was actively bending the light."[1]

However, the mathematics of high-dimensional geometry complicate this assumption. A neural network with 12,288 dimensions, such as the base architecture of GPT-4, contains vastly more orthogonal directions than a human can visualize. When a linear probe attempts to draw a boundary in this space, it makes geometric compromises that distort the underlying data structure.[1][2]

The mechanics of rank distortion

A linear probe operates by finding a hyperplane that cleanly separates activations associated with a specific concept from those that are not. If researchers want to know if a model recognizes deception, they train a probe to find the geometric plane where all deceptive examples fall on one side. The probe then assigns weights based on this distance.

The distortion occurs because standard linear probes are optimized purely for classification accuracy, without constraints on the mathematical rank of their weight matrices. In a 2024 study published in the Journal of Machine Learning Research, researchers demonstrated that unconstrained probes exploit spurious correlations in high-dimensional noise. The probe stretches and rotates the activation space to maximize its margin.

To visualize this, imagine trying to cast a shadow of a complex three-dimensional object onto a two-dimensional wall. By rotating the light source, you can make a cylinder look like either a circle or a rectangle. A standard linear probe effectively rotates the mathematical light source until the resulting shadow perfectly matches the target concept.[2]

This geometric warping forces orthogonal features to cast shadows onto the same neuron. If a neuron genuinely represents syntax formatting, but the probe's distortion matrix rotates the deception feature into the same subspace, the probe registers a false positive. The neuron appears polysemantic to the observer, but it is actually monosemantic in the model's native geometry.[1]

"The probe is essentially acting like a funhouse mirror, bending the activation space until the target concept aligns with whatever neurons happen to have high variance," Rostova explains in her 2026 preprint. "This means the polysemanticity crisis that has defined mechanistic interpretability for three years is partly an artifact of flawed instrumentation."[1]

Quantifying the misattribution error

When researchers correct for these geometric distortions, the apparent polysemanticity of large language models drops precipitously. Standard linear probes applied to a 7-billion-parameter model typically report that 78 percent of neurons respond to more than three distinct concepts. This high rate of overlap led early safety teams to conclude that manual circuit analysis would never scale.[1][2]

However, applying a rank-penalized probe—which mathematically restricts the classifier from warping the activation geometry—changes the results entirely. In a comprehensive 2026 evaluation of the Llama-3 architecture, rank-corrected probes found that only 12 percent of the same neurons exhibited true polysemantic behavior. The remaining 66 percent of the overlap was entirely generated by projection artifacts.[1][2]

This massive discrepancy explains why early attempts to edit polysemantic neurons often catastrophically degraded model performance. If a researcher suppresses a neuron because a distorted probe claims it represents bias, they are actually destroying the orthogonal, monosemantic feature that the neuron genuinely encodes. The model loses capabilities because the safety intervention was guided by a hallucinated map.[1]

The discovery of rank distortion also resolves a lingering paradox in the superposition literature. Models trained with high sparsity penalties still appeared highly polysemantic when measured with standard probes, baffling researchers who expected the sparsity to force monosemantic representations. The models were indeed learning clean features, but the unconstrained probes were re-entangling them during measurement.

By analyzing the singular value decomposition of the probe's weight matrix, mathematicians can now quantify exactly how much distortion is occurring. A probe with a highly skewed singular value spectrum is actively compressing the space, while a flat spectrum indicates a faithful geometric mapping. This diagnostic check has become a mandatory step in modern interpretability pipelines.

Sparse autoencoders isolate features

To bypass the flaws of linear probes, the interpretability community has rapidly pivoted to sparse autoencoders. An autoencoder is a secondary neural network trained to reconstruct the target model's activations using a much larger, but strictly sparse, set of artificial neurons. Because it expands the dimensional space rather than projecting it down, it avoids rank compression entirely.[2]

In late 2025, Anthropic successfully applied a 34-million-feature sparse autoencoder to their Claude 3 Sonnet model. This massive expansion isolated highly specific, monosemantic concepts like transit infrastructure and immunology. Unlike linear probes, the autoencoder does not require labeled data for a specific concept, allowing it to unsupervisedly disentangle the activation space into its true constituent parts.

When researchers compare autoencoder feature attributions against standard linear probe results, the extent of the probe's distortion becomes undeniable. The autoencoder reveals that the concepts the linear probe crammed into a single polysemantic neuron are actually distributed across hundreds of distinct, monosemantic directions. The linear probe was simply incapable of resolving that fine-grained geometry.[1]

"We spent years trying to untangle a knot that our own tools were tying," notes Dr. David Bau, a computer scientist at Northeastern University, in a recent symposium on representation engineering. "By abandoning unconstrained linear probes, researchers can finally map the true causal structure of frontier models without the interference of projection artifacts."

Implications for model auditing

The recognition of linear probe rank distortion fundamentally alters how regulators and safety institutes audit artificial intelligence. Under the European Union's AI Act, companies deploying high-risk models must demonstrate an understanding of their systems' internal decision-making. Relying on standard linear probes to fulfill this requirement would produce legally binding documentation filled with mathematical hallucinations.[2]

Safety institutes are now updating their evaluation frameworks to mandate rank-corrected diagnostics. The UK AI Safety Institute recently published guidelines requiring developers to use either sparse autoencoders or strictly regularized probes when submitting mechanistic interpretability evidence. This ensures that any claims about a model's internal safeguards are based on its actual geometry, not instrumental artifacts.

Furthermore, this geometric clarity accelerates the development of concept erasure techniques. If researchers can accurately isolate the true direction of a dangerous capability, they can mathematically subtract that vector from the model's weights. Rank-corrected mapping ensures this erasure removes only the targeted threat, without accidentally lobotomizing orthogonal skills.[1]

The financial implications for frontier AI labs are also substantial. Training sparse autoencoders requires massive compute overhead, often consuming up to 20 percent of the original model's training budget. By contrast, rank-penalized linear probes offer a computationally cheap alternative that still avoids the worst geometric distortions.[2]

This efficiency allows smaller research teams to conduct rigorous interpretability work without requiring supercomputer access. Academic laboratories can now audit billion-parameter models on standard hardware by simply swapping their linear probe implementations for rank-penalized variants. The democratization of accurate interpretability tools accelerates the broader field of AI safety.[1]

The mathematical bounds of superposition

Even with rank-corrected tools, the phenomenon of superposition does not completely disappear from artificial neural networks. Models still compress more features than they have dimensions, but they do so through structured, predictable geometries rather than chaotic polysemantic tangles. When measured accurately, these compressed representations resemble highly symmetrical polyhedra in the activation space.[2]

Researchers at the Swiss Federal Institute of Technology demonstrated in early 2026 that true superposition only occurs when features are mutually exclusive. If a model knows that a token cannot be both a noun and a verb simultaneously, it can safely pack those two concepts into the same subspace. Rank-corrected probes can successfully identify these antipodal pairings.

This structural revelation provides a clear roadmap for designing more interpretable architectures from the ground up. If engineers understand the exact geometric conditions under which models compress features, they can modify the training objective to encourage orthogonal, monosemantic representations. The goal is no longer to untangle a black box, but to build a glass box that naturally resists rank distortion.[2]