arXiv cs.CL· Nathan Godey, Yoav Artzi·· 3 天前AI 评分26
Copy-Paste Language Models：为 LLM 增加复制粘贴头提升上下文复制能力
Copy-Paste Language Models
AI 导读
研究者提出 Copy-Paste Language Models（CPLMs），在传统语言建模（LM）头之外增加一个轻量级 Copy-Paste 头，通过显式建模提升 LLM 的上下文复制能力。
正文 · 原文
Abstract:Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.19247 [cs.CL] |
| (or arXiv:2610.19247v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19247 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nathan Godey [view email]
[v1]
Tue, 13 Oct 2026 22:03:18 UTC (750 KB)
来源：arXiv cs.CL · arxiv.org