Inter-Layer Structural Encoders (ILSE) use Cayley graphs to aggregate features from all internal LLM layers, improving accuracy by up to 44% over final-layer-only predictions.
March 25, 2026
Original Paper
Improving LLM Predictions via Inter-Layer Structural Encoders
arXiv · 2603.22665
The Takeaway
It challenges the standard practice of only using the final layer for token representations, proving that critical task-relevant information is often trapped in intermediate layers. This approach allows smaller models to outperform much larger counterparts by more effectively utilizing their internal architectural depth.
From the abstract
The standard practice in Large Language Models (LLMs) is to base predictions on the final-layer token representations. Recent studies, however, show that intermediate layers encode substantial information, which may contain more task-relevant features than the final-layer representations alone. Importantly, it was shown that for different tasks, different layers may be optimal. In this work we introduce Inter-Layer Structural Encoders (ILSE), a powerful structural approach to learn one effective