Successfully trains a 0.9B parameter pure Spiking Neural Network (SNN) from scratch for language modeling, achieving performance without Transformer distillation.
arXiv · March 18, 2026 · 2603.16148
The Takeaway
Proves that SNNs—which are significantly more energy-efficient on neuromorphic hardware—can scale to near-billion parameter LLM tasks, a major milestone for low-power embodied AI.
From the abstract
We ask whether a pure spiking backbone can learn large-scale language modeling from random initialization, without Transformer distillation. We introduce NeuronSpark, a 0.9B-parameter SNN language model trained with next-token prediction and surrogate gradients. The model combines selective state-space spiking dynamics, leakage-current inter-layer communication, PonderNet adaptive timesteps, fused Triton PLIF kernels, and stabilization techniques (residual centering, lateral-inhibition normaliza