多 Token 预测与替代分词让语言模型在大规模图上学会隐式多跳搜索

Multi-Token Prediction and Alternative Tokenization Improve Learning Latent Multi-Hop Graph Search at Scale

AI 导读

语言模型在图上学习最短路径任务时随规模扩大而失败，研究指出原因在于监督信号与分词方式两方面。作者从理论上证明并实验验证了多 Token 预测（MTP）的有效性，且该收益与图的拓扑结构无关；同时引入替代分词方案，证明两个原因彼此正交。结合这两种方法，模型突破了此前工作中的规模瓶颈，在远超以往限制的规模上稳定学会求解该任务。

正文 · 原文

View PDF HTML (experimental)

Abstract:Given an unseen graph, a start node, $s$, and an end node, $t$, can a language model generate the shortest path from $s$ to $t$? Prior work has found that models fail to learn this task as the graphs scale. We show that this failure has two causes: the supervision signal and the tokenization. First, we theoretically establish and empirically demonstrate the effectiveness of multi-token prediction (MTP), and show that the benefits of MTP are topologically agnostic. Second, we introduce alternative tokenizations and show that the two causes are orthogonal. Using these methods, we break through the scaling wall observed by prior work, consistently and stably learning to solve the task at scales far beyond previous limits.

Comments:	Don't be intimidated by the page length :)
Subjects:	Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as:	arXiv:2610.31439 [cs.LG]
	(or arXiv:2610.31439v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2610.31439 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Arvid Frydenlund [view email]
[v1] Thu, 22 Oct 2026 15:35:50 UTC (3,033 KB)

来源：arXiv cs.CL · arxiv.org