arXiv cs.AI· Chun-Hao Yang, Bo-Han Feng, Tzu-Yuan Lai, Yan-Yu Chen, Yin-Kai Dean Huang, Shou-De Lin·· 3 天前AI 评分24
超越下一 token 预测：Max(MI(S;t)) 如何填补 LLM 训练的互信息缺口
Training LLMs Beyond Next Token Prediction - Filling the Mutual Information Gap
AI 导读
研究提出基于互信息的确定性启发式方法 Max(MI(S;t))，通过优先预测信息量丰富的目标 token 来组织训练时的目标顺序，在算术推理、多标签文本分类和文本生成任务中常优于标准 NTP 及 reverse order、TF-IDF、条件概率选择等基线。该方法在 GPT 风格模型及预训练 Llama/Qwen 变体上验证，无需改动模型架构或自回归解码。
正文 · 原文
Abstract:Autoregressive language models are usually trained with next-token prediction under a fixed left-to-right target order. This paper studies target-token organization as a training-time design choice across arithmetic reasoning, multi-label text classification, and text generation. Controlled order studies show that the default NTP order can be far from the best-performing tested organization, while fixed hand-crafted orders such as reverse order do not generalize across tasks. To address this, Max(MI(S;t)) is introduced as a deterministic mutual-information-based heuristic that prioritizes information-rich target tokens. For fixed-length outputs, it constructs task-specific prediction orders; for open-vocabulary text generation, the same principle is applied through lightweight prefix augmentation. Experiments with GPT-style models and pretrained Llama/Qwen variants show that Max(MI(S;t)) often improves over standard NTP and simple baselines such as reverse order, TF-IDF, and conditional-probability selection, without changing model architecture or autoregressive decoding.
| Comments: | AACL-IJCNLP 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2511.00198 [cs.CL] |
| (or arXiv:2511.00198v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2511.00198 arXiv-issued DOI via DataCite |
Submission history
From: Chun-Hao Yang [view email]
[v1]
Fri, 31 Oct 2025 18:59:29 UTC (189 KB)
[v2]
Tue, 13 Oct 2026 06:48:26 UTC (415 KB)
来源：arXiv cs.AI · arxiv.org