Frontier models' reasoning steps are largely 'decorative' and do not causally determine the final answer in most tasks.
March 25, 2026
Original Paper
When AI Shows Its Work, Is It Actually Working? Step-Level Evaluation Reveals Frontier Language Models Frequently Bypass Their Own Reasoning
arXiv · 2603.22816
The Takeaway
This study reveals that removing reasoning steps from models like GPT-4 or Claude Opus often leaves the answer unchanged, challenging the assumption that Chain-of-Thought (CoT) reflects a causal inference process. It introduces a cheap, black-box 'step-level evaluation' that practitioners can use to audit the actual faithfulness of AI reasoning.
From the abstract
Language models increasingly "show their work" by writing step-by-step reasoning before answering. But are these reasoning steps genuinely used, or decorative narratives generated after the model has already decided? Consider: a medical AI writes "The patient's eosinophilia and livedo reticularis following catheterization suggest cholesterol embolization syndrome. Answer: B." If we remove the eosinophilia observation, does the diagnosis change? For most frontier models, the answer is no - the st