OBD-LLM uses second-order Hessian information to achieve 20-40% better low-rank decomposition accuracy than the current state-of-the-art SVD-LLM.
April 2, 2026
Original Paper
Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation
arXiv · 2604.00821
The Takeaway
It provides a closed-form solution for weight matrix compression that considers both input and output layer information. This enables significantly higher compression ratios for LLMs with lower performance degradation during post-training optimization.
From the abstract
Low-rank decomposition has emerged as an important problem in Large Language Model (LLM) fine-tuning and inference. Through Singular Value Decomposition (SVD), the weight matrix can be factorized into low-rank spaces optimally. Previously, a common practice was to decompose the weight in the activation-whitened space, and then achieve satisfying results. In this work, we propose Optimal Brain Decomposition LLM (OBD-LLM), which studies the decomposition problem in the model space by utilizing sec