arXiv cs.CL· Chenyu Hui·· 12 小时前AI 评分34
通过主动回顾学习提升大语言模型长上下文理解能力
Towards robust long-context understanding of large language model via active recap learning
AI 导读
研究者提出主动回顾学习（ARL）框架，通过持续预训练中的定向序列构建和推理时的回顾式摘要，让模型重新访问并总结前文内容。ARL 在 RULER 上最高提升 129.7%，在 LongBench 上提升 11–13%。该方法基于损失差识别关键 token 并定位相关前序段落，使模型在推理时自主生成回顾摘要，形成跨段落的递归记忆机制。
正文 · 原文
Abstract:In this paper, we propose active recap learning (ARL), a framework for enhancing large language model (LLM) in understanding long contexts. ARL enables models to revisit and summarize earlier content through targeted sequence construction during continued pretraining and retrospective summarization at inference. First, we identify key tokens in prepared long context based on loss gaps between long and short forward contexts and find the most relevant preceding paragraphs, then summarize them using an LLM. Second, ARL equips models with the ability to autonomously generate and utilize these retrospective summaries during inference, thereby establishing a recursive memory mechanism across paragraphs. Experimental results show substantial gains, with ARL achieving an improvement of up to 129.7\% on RULER and 11--13\% on LongBench. Overall, ARL offers a simple yet effective continued pretraining-based approach to strengthen long-context understanding, advancing scalable memory augmentation in LLM. Code is available at \href{this https URL}{CODE}
| Comments: | 6 pages |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| MSC classes: | 68T35 |
| ACM classes: | F.2.2; I.2.7 |
| Cite as: | arXiv:2601.13734 [cs.CL] |
| (or arXiv:2601.13734v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2601.13734 arXiv-issued DOI via DataCite |
Submission history
From: Chenyu Hui [view email]
[v1]
Tue, 20 Jan 2026 08:42:04 UTC (116 KB)
[v2]
Fri, 23 Oct 2026 11:17:08 UTC (977 KB)
来源：arXiv cs.CL · arxiv.org