arXiv cs.CL· Haoran Zheng, Chen Shani·· 12 小时前AI 评分34
Conceptizer：下一 token 预测并非你所需的全部
Conceptizer: Next-Token Prediction Is Not All You Need
AI 导读
研究提出 Conceptizer 训练框架，在下一 token 预测之外增加语义奖励：用约 9K 个由上下文同义词构成的固定概念词表，将每个下一 token 分布映射为概念分布，把目标语义的奖励分摊到多种可能措辞上，且不增加模型参数。
正文
Abstract:Next-token prediction (NTP) rewards only the token that occurs next, penalizing every alternative. But natural language often offers many ways to express the same meaning. We introduce Conceptizer, a training framework that extends next-token prediction with a reward for meaning. Conceptizer uses a fixed, model-agnostic vocabulary of ~9K concepts, built from contextual synonyms, as units of meaning alongside tokens as units of text. It maps each next-token distribution to a concept distribution and shares the reward for the target meaning across its possible wordings. This provides a concept-level training signal without adding model parameters. Applied to 3 model families at 1B-8B parameters, Conceptizer achieves the lowest held-out NTP perplexity at concept-target positions and assigns the highest probability to contextual synonyms, against all 3 baselines. It also improves downstream performance on knowledge and sentiment tasks while remaining comparable on commonsense. These results show that language models can be directly optimized to predict meaning, not just the next token.
| Comments: | 35 pages, 10 figures, 16 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.33027 [cs.CL] |
| (or arXiv:2610.33027v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33027 arXiv-issued DOI via DataCite |
Submission history
From: Haoran Zheng [view email]
[v1]
Sun, 11 Oct 2026 08:59:07 UTC (408 KB)
来源：arXiv cs.CL · arxiv.org