arXiv cs.CL· Manas Mehta, Fangcong Yin, Greg Durrett·· 12 小时前AI 评分32
Randomized YaRN 提升长上下文推理的长度泛化能力
Randomized YaRN Improves Length Generalization for Long-Context Reasoning
AI 导读
Randomized YaRN 将 YaRN 位置外推与随机位置编码、长度课程结合，在短上下文数据训练时给 token 分配采样自更大位置范围的 YaRN 位置编码，使模型接触分布外位置表示。在 BABILong、MRCR 和 LongBench v2 三个长上下文推理基准上，该方法在 16K 至 128K 上下文长度上持续优于标准微调，增益在远超分布外的长度上最大。
正文 · 原文
Abstract:Large language models (LLMs) are typically pretrained on short sequences and then extended to work on longer sequences with additional training. However, such LLMs still struggle to further generalize to very long sequences. We propose Randomized YaRN, a training method that improves length generalization by combining YaRN-based positional extrapolation with randomized positional encoding and a length curriculum. During training on short context data, tokens are assigned YaRN positional encodings sampled from a larger position range, exposing the model to out-of-distribution positional representations even on short-context inputs. We evaluate Randomized YaRN on three challenging long-context reasoning benchmarks, BABILong, Multi-Round Coreference Resolution (MRCR), and LongBench v2. When training on data with short context, Randomized YaRN consistently improves reasoning performance on context lengths from 16K to 128K and outperforms standard fine-tuning, with the largest gains appearing at far out-of-distribution lengths. Our results suggest that progressively exposing models to OOD positional distributions provides an effective recipe for generalizable long-context reasoning.
| Comments: | EMNLP 2026 Findings |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.23687 [cs.CL] |
| (or arXiv:2606.23687v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.23687 arXiv-issued DOI via DataCite |
Submission history
From: Fangcong Yin [view email]
[v1]
Mon, 22 Jun 2026 17:59:53 UTC (470 KB)
[v2]
Sun, 30 Aug 2026 21:36:23 UTC (473 KB)
[v3]
Tue, 20 Oct 2026 16:46:46 UTC (473 KB)
来源：arXiv cs.CL · arxiv.org