HoloByte is a tokenizer-free framework that projects byte sequences into a continuous hyperspherical manifold to bypass the morphological limits of discrete tokens.
March 19, 2026
Original Paper
HoloByte: Continuous Hyperspherical Distillation for Tokenizer-Free Modeling
arXiv · 2603.16917
The Takeaway
It addresses the fundamental bottleneck of byte-level attention by reducing exact attention complexity from O(N^2) to O((N^2/W^2)D + ND^2). This allows for continuous optimization of raw data without artificial vocabulary boundaries, potentially changing how we model multi-modal or low-resource languages.
From the abstract
Sequence modeling universally relies on discrete subword tokenization to circumvent the $\mathcal{O}(N^2)$ computational intractability of native byte-level attention. However, this heuristic quantization imposes artificial morphological boundaries, enforces vocabulary dependence, and fractures the continuity of the optimization landscape. To resolve this dichotomy, we introduce \textbf{HoloByte}: a strictly tokenizer-free framework utilizing Continuous Hyperspherical Distillation. HoloByte part