arXiv cs.CL· Wanyang Hong, Zhaoning Zhang, Yi Chen, Libo Zhang, Baihui Liu, Linbo Qiao, Zhiliang Tian, Dongsheng Li·· 17 小时前AI 评分32
REA：面向对话 LLM 的角色感知启发式情景注意力框架
Role-aware Heuristic Episodic Attention for Conversational LLMs
AI 导读
针对多轮对话中指令与信息随上下文增长而衰减的问题，研究者提出 REA（Role-aware Heuristic Episodic Attention）上下文管理框架，将全局约束存入专用前缀的指令记忆，并对历史轮次按启发式选择原文、压缩表示或省略。
正文 · 原文
Abstract:Large language models often lose track of persistent instructions and relevant information as multi-turn conversations grow. We study this cumulative contextual decay through three related failure modes: attention pollution, dilution, and drift. We propose REA(Role-aware Heuristic Episodic Attention), a context-management framework that assigns different persistence and representation policies to instructions and episodic interactions. Instructional Memory retains identified global constraints in a dedicated prefix. Episodic Memory preserves user inputs and compresses model replies, while heuristic retrieval selects raw text, compressed representations, or omission for each historical turn. On Long-MT-Bench+, REA improves the judge score from 6.32 to 7.36 on a 10-point scale, a 16.5% relative gain over the Vanilla baseline, and reduces average latency by 2.91x. Additional evaluations show aggregate gains on three backbones spanning 1.7B-7B parameters and on Chinese and English role-playing tasks. These results support role-aware context management as a practical approach to maintaining conversational continuity and instruction adherence.
| Comments: | 16 pages, 6 figures |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2512.06869 [cs.CL] |
| (or arXiv:2512.06869v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2512.06869 arXiv-issued DOI via DataCite |
Submission history
From: Wanyang Hong [view email]
[v1]
Sun, 7 Dec 2025 14:50:03 UTC (1,281 KB)
[v2]
Mon, 26 Oct 2026 05:04:06 UTC (570 KB)
来源：arXiv cs.CL · arxiv.org