The study provides a formal link showing that internal 'world model' representations in transformers are a direct byproduct of the predictive geometry of the training data.
arXiv · March 18, 2026 · 2603.16689
The Takeaway
It offers a concrete, traceable explanation for why next-token predictors develop low-dimensional latent maps of their environment. This moves the field away from purely heuristic interpretations of world models toward a mathematically grounded understanding of structural internalization.
From the abstract
Next-token predictors often appear to develop internal representations of the latent world and its rules. The probabilistic nature of these models suggests a deep connection between the structure of the world and the geometry of probability distributions. In order to understand this link more precisely, we use a minimal stochastic process as a controlled setting: constrained random walks on a two-dimensional lattice that must reach a fixed endpoint after a predetermined number of steps. Optimal