Why Interpretability Stalled - and How It Gets Unstuck
For more than a decade, interpretability has been one of the most intellectually compelling areas of machine learning. It promised something rare: the ability to look inside models that were rapidly becoming more capable, more opaque, and more consequential.
And yet, despite serious effort from some of the best research groups in the world, interpretability has struggled to translate into tools that engineers actually rely on in production. The field hasn't failed - but it has stalled.
This post is an attempt to explain why. Not in the abstract, and not politely. The goal is to name the real bottlenecks that slowed progress, to acknowledge critiques that are largely correct, and to outline what has quietly started to change.
This is not a manifesto for interpretability as curiosity. It is an argument for interpretability as an engineering discipline.
The uncomfortable truth: the field optimized for insight, not reliability
Interpretability research has historically been rewarded for producing insightful artifacts. A clean neuron visualization. A compelling story about how a circuit implements a behavior. A dashboard that makes a model's internals feel legible.
Those artifacts matter. But they were often mistaken for something stronger than they were.
In practice, many interpretability results collapse under mild perturbations. Change the prompt. Change the phrasing. Change the distribution. Change the checkpoint. Suddenly the explanation no longer holds, or worse, still looks plausible while silently becoming wrong.
This isn't a minor flaw. It is the defining reason interpretability stalled.
Engineering disciplines don't stall because they lack ideas. They stall when results can't be trusted outside carefully curated conditions.
One-prompt explanations don't survive the real world
A large fraction of interpretability work is demonstrated on a small number of hand-picked prompts. That's understandable: interpretability is expensive, and clarity matters.
But the moment an explanation only holds for a narrow slice of inputs, it stops being an explanation and starts being a demo.
This problem is not unique to language models. The saliency literature in vision learned this the hard way. Researchers showed that visually compelling explanations could remain nearly unchanged even when model weights were randomized - meaning the "explanation" was not actually tied to the computation being performed.
Language models inherited the same failure mode, just in a subtler form. Explanations often sound reasonable because humans are very good at filling in gaps. That does not mean the explanation is faithful.
An explanation that does not survive paraphrase is not a mechanism. It is a story.
The one-layer fallacy
Another quiet trap is the assumption that explanations can be localized cleanly to a single neuron, head, or layer.
Modern transformers do not work that way.
The best mechanistic work in the field - such as the indirect object identification circuit in GPT-2 - was compelling precisely because it avoided this trap. It showed a distributed mechanism spanning multiple components, and it came with explicit causal tests.
But those successes were often misinterpreted as evidence that "we found the neuron" was the goal. In reality, they demonstrated the opposite: reasoning is distributed, entangled, and fragile under basis changes.
Work on superposition made this explicit. When models compress many concepts into limited representational space, individual units inevitably become polysemantic. In that regime, the question "what does this neuron mean?" is ill-posed.
Interpretability stalled partly because the field kept asking questions that the models were not structured to answer cleanly.
The benchmark vacuum and the rise of hype cycles
Interpretability lacks what every mature scientific field eventually converges on: shared benchmarks that force hard comparisons.
Too many interpretability results are evaluated relative to other interpretability methods, rather than against strong non-interpretability baselines. When the baselines are weak, progress looks impressive even when it has little practical impact.
This is exactly the critique articulated by the Martian interpretability challenge. Their framing is blunt but accurate: without benchmarks tied to ground-truth mechanisms or real downstream outcomes, interpretability risks becoming self-referential.
Negative results matter here. DeepMind's finding that sparse autoencoders underperformed simple linear probes for certain safety-relevant classification tasks was not a failure of SAE research - it was a corrective. It forced the field to confront the reality that interpretability methods must earn their keep.
Engineering disciplines advance when they are forced to compete with simple, effective alternatives.
Closed models broke the research-to-production bridge
Another reason interpretability stalled is structural.
A large amount of interpretability tooling assumes access that does not exist in many real deployments: internal activations, patching hooks, reproducible checkpoints, unlimited experimentation.
Those assumptions are reasonable for open research models. They break down quickly in enterprise and regulated environments.
Even when internal access exists, the workflows are often exploratory rather than operational. Tools like TransformerLens unlocked enormous research power, but power alone does not create reliability. Exploratory tools become production tools only when they are paired with evaluation discipline, versioning, and failure handling.
Interpretability stalled because it lived too long as analysis rather than infrastructure.
Why "pretty dashboards" aren't evidence
Visualization is seductive. A good interface can make complex systems feel understandable.
But visualization without falsification is dangerous.
Many interpretability dashboards optimize for insight density, not error detection. They surface interesting features, but they rarely tell you when those features stop working, drift, or become misleading.
This is where skepticism becomes justified. If an explanation cannot tell you when it is wrong, it cannot support real decisions.
Dashboards are interfaces. Evidence is measurement.
The four questions that actually matter
A skeptic can summarize the state of interpretability with four questions:
- Is it mechanistic, or just correlational?
- Does it help anyone do something better?
- Does it generalize beyond toy settings?
- Can it scale with model size and deployment reality?
For years, interpretability struggled to answer all four at once.
What's changed recently is not a single breakthrough, but a shift in orientation.
What finally starts to unstick the field
Interpretability begins to work when it stops trying to explain everything and starts trying to support decisions.
That shift has several consequences.
Evaluation becomes central. Interpretability methods are no longer judged by how compelling they look, but by how they perform under perturbation, distribution shift, and model updates.
Artifacts become standardized. Instead of ad-hoc explanations, teams converge on repeatable objects: reports that capture how a feature behaves, how stable it is, what breaks it, and how it should be monitored over time.
Negative controls become mandatory. If a method cannot fail in obvious ways when it should, it cannot be trusted when it succeeds.
Interpretability becomes update-aware. Models change. Fine-tuning happens. Data drifts. Any explanation that assumes a static model is already obsolete.
This is why newer work increasingly emphasizes versioning, drift measurement, and lifecycle integration. Interpretability that cannot survive model updates cannot survive production.
Scaling without pretending we understand everything
Scalability is the hardest problem. Circuit analysis does not scale linearly with model size. Dictionary-learning approaches like sparse autoencoders scale computationally, but introduce new validation challenges.
Open efforts like Gemma Scope matter because they shift the field from bespoke SAE training to shared infrastructure. They reduce friction and make comparison possible. They do not solve interpretability - but they make systematic progress possible.
Similarly, large-scale SAE work at frontier labs demonstrates both promise and limits. Extracting millions of features is impressive. Validating them meaningfully is the real bottleneck.
The field moves forward when it acknowledges this tradeoff instead of denying it.
Interpretability doesn't become useful by being more clever - it becomes useful by being boring
This is the final and perhaps most important point.
Interpretability stalled when it tried to be dazzling.
It starts working when it becomes boring.
Boring means repeatable. Boring means measurable. Boring means compatible with audits, incident reviews, and engineering workflows. Boring means admitting uncertainty and surfacing failure modes.
The future of interpretability is not a single beautiful explanation. It is a control and evidence layer that sits alongside training and evaluation, quietly answering questions like:
What changed?
What broke?
What should we trust?
What needs intervention?
When interpretability answers those questions reliably, it stops stalling.
It becomes infrastructure.