Your LLM knows it's about to lie to you, but it's mathematically incapable of stopping itself.
We see LLMs express 'uncertainty' and think they're becoming self-aware, but this paper reveals the 'metacognitive gap.' While models can verbalize that they are unsure about an answer, they are consistently unable to use that information to correct their actual output. The 'self-awareness' is just a linguistic pattern, disconnected from the reasoning engine. For practitioners, this means 'confidence scores' are currently a useless indicator for real-time error correction. We need to bridge this gap before AI can truly 'double-check' its own work.
FINAL Bench: Measuring Functional Metacognitive Reasoning in Large Language Models
SSRN · 6280258
Existing AI benchmarks (MMLU, HumanEval, GPQA) measure only final-answer accuracy, neglecting the core hallmark of expert-level intelligence: the ability to detect and correct one's own reasoning errors. Although partial metacognitive behaviors have been observed in large reasoning models (DeepSeek-AI, 2025; Wan et al., 2025), no unified benchmark exists for their systematic measurement. We introduce FINAL Bench (Frontier Intelligence Nexus for AGI-Level Verification), the first benchmark for ev