arXiv cs.CL· Kohei Kajikawa, Lin Ai, Tatsuki Kuribayashi, Ethan Gotlieb Wilcox·· 9 天前AI 评分22
用 GPT-2 建模上下文窗口与指代关系对句子理解的影响
Using LMs to Model the Effects of Context and Coreference during Sentence Comprehension
AI 导读
研究系统改变 GPT-2 的上下文窗口大小，在四个大规模英语阅读时间数据集上发现 U 型关系：受限上下文（< 20 tokens）能捕捉局部记忆限制，但扩展上下文（500–1,000 tokens）整体心理语言学拟合度最高。
正文 · 原文
Abstract:Language models (LMs) are often used as a tool to model human language processing. Recent studies suggest that severely restricting LMs' context window improves their fit to human psycholinguistic data by simulating human working memory constraints. However, it is possible that this strict memory-decay approach overlooks humans' reliance on long-range structural representations, such as discourse structre. In this work, we systematically vary the context window size of GPT-2 across four large-scale naturalistic English reading-time datasets and observe a U-shaped relationship: Although restricted contexts (< 20 tokens) successfully capture local memory limitations, expanded contexts (500--1,000 tokens) ultimately yield the highest overall psycholinguistic fit. To investigate the mechanism driving this benefit, we conduct a counterfactual inference-time experiment that disrupts cross-sentential entity chains by pronominalizing repeated discourse entities. Obscuring these structural linkages significantly degrades the predictive power of larger context windows by 20% to 40%. Our experiments demonstrate that tracking long-range coreference relations is one important factor for the alignment between LM surprisal and human reading behavior, and approximate the extent to which human comprehenders use global discourse relations during language processing.
| Comments: | EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.32119 [cs.CL] |
| (or arXiv:2609.32119v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.32119 arXiv-issued DOI via DataCite |
Submission history
From: Kohei Kajikawa [view email]
[v1]
Sat, 26 Sep 2026 00:43:56 UTC (580 KB)
[v2]
Thu, 8 Oct 2026 01:48:04 UTC (580 KB)
来源：arXiv cs.CL · arxiv.org