Engineered modularity via per-layer supervision solves the 'Hydra effect,' allowing for the surgical control of specific model behaviors.
arXiv · March 20, 2026 · 2603.18029
The Takeaway
Standard Transformers exhibit high redundancy that makes identifying and controlling specific circuits nearly impossible. By introducing architectural gates and independent layer gradients, this work enables 'control leverage' where scaling specific attention heads produces predictable behavioral changes, moving interpretability from correlation to causal control.
From the abstract
Transformers resist surgical control. Ablating an attention head identified as critical for capitalization produces minimal behavioral change because distributed redundancy compensates for damage. This Hydra effect renders interpretability illusory: we may identify components through correlation, but cannot predict or control their causal role. We demonstrate that architectural interventions can expose hidden modularity. Our approach combines dual-stream processing separating token and contextua