Theoretical and empirical evidence suggests that the 'Key' mechanism in Attention may be redundant, proposing a 'QV' paradigm that simplifies Transformer architectures.
arXiv · March 18, 2026 · 2603.15665
The Takeaway
By identifying optimization trajectories that lead to QV-only attention, this work paves the way for more efficient model architectures with fewer parameters and lower compute requirements during both training and inference.
From the abstract
Starting from first principles and a linguistic perspective centered on part-of-speech (POS) and syntactic analysis, this paper explores and derives the underlying essence of the Query-Key-Value (QKV) mechanism within the Transformer architecture. Based on this theoretical foundation, we provide a unified explanatory framework for the efficacy of contemporary architectures, including MQA, GQA, and MLA, while identifying their inherent trade-offs and potential optimization trajectories. We introd