Switches the training objective from hard Next-Token Prediction to predicting 'concepts' (sets of semantically related tokens).
April 1, 2026
Original Paper
Concept Training for Human-Aligned Language Models
arXiv · 2603.29123
The Takeaway
It addresses the 'mutually exclusive' target problem in NTP, where similar words compete for probability. Training on semantic clusters improves human alignment and reduces perplexity on meaningful content, suggesting a path beyond the limitations of strict token-matching.
From the abstract
The next-token prediction (NTP) objective trains language models to predict a single continuation token at each step. In natural language, however, a prefix can be continued in many valid ways, and even similar meanings may differ in surface form. For example, the sentence ``this website is safe to \underline{browse}'' could plausibly continue with words such as browse, search, visit, surf, or navigate. While standard NTP training treats these alternatives as mutually exclusive targets, we explo