Reveals that RLVR-driven reasoning improvements in LLMs are the result of highly sparse changes to a tiny fraction of 'critical' token distributions.
March 25, 2026
Original Paper
Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
arXiv · 2603.22446
The Takeaway
The study shows that injecting a small set of RL-sampled tokens into base generations can recover nearly all performance gains. This suggests that LLM reasoning 'breakthroughs' are concentrated in specific logical pivot points rather than a broad stylistic shift, offering a new target for efficient fine-tuning.
From the abstract
Reinforcement learning with verifiable rewards (RLVR) has significantly improved reasoning in large language models (LLMs), yet the token-level mechanisms underlying these improvements remain unclear. We present a systematic empirical study of RLVR's distributional effects organized around three main analyses: (1) token-level characterization of distributional shifts between base and RL models, (2) the impact of token-level distributional shifts on sequence-level reasoning performance through cr