arXiv cs.AI· Yukun Huang, Sanxing Chen, Jian Pei, Manzil Zaheer, Bhuwan Dhingra·· 3 天前AI 评分23
Cite Pretrain：让大语言模型无需检索即可完成知识引用
Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models
AI 导读
研究者提出 Cite Pretrain 方法，让 LLM 无需推理时检索即可引用其（持续）预训练中的文档，并推出 CitePretrainBench 基准，涵盖 Wikipedia、Common Crawl、arXiv 及未见文档。
正文 · 原文
Abstract:Large language models can answer questions correctly and still hallucinate their citations. Most citation systems address this by retrieving documents at inference time. Retrieval adds latency and depends on external infrastructure. Retrieval noise can also affect the answer. We study whether LLMs can cite documents from their (continual) pretraining without test-time retrieval. We introduce CitePretrainBench to evaluate this setting. It includes Wikipedia, Common Crawl, and arXiv as real-world corpora and also includes novel, unseen documents. The benchmark includes short-form citation tasks for a single fact and long-form tasks for multiple facts. Continual-pretraining first binds factual knowledge to a persistent identifier for each document. Instruction tuning then teaches the model to put these identifiers in its answers as citations. Passive Indexing appends an identifier at the end of each document. Synthetic data used in Active Indexing restate facts in diverse, compositional forms. We train two mappings together. These mappings are source$\to$fact and fact$\to$source. The first direction involves the model generating content given a cited source. In the second, it learns to attribute its own answers when facts are paraphrased or combined. On Qwen-2.5-7B and 3B, Active Indexing improves citation precision over Passive Indexing on every evaluated citation task, with gains of up to 30.2 percentage points. In the scaling ablation, citation performance continues to improve as we add augmented data, including at 16 times the original token count. Combining internal and external citations also improves robustness to retrieval noise.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2506.17585 [cs.AI] |
| (or arXiv:2506.17585v4 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2506.17585 arXiv-issued DOI via DataCite |
Submission history
From: Yukun Huang [view email]
[v1]
Sat, 21 Jun 2025 04:48:05 UTC (614 KB)
[v2]
Tue, 28 Oct 2025 18:06:24 UTC (683 KB)
[v3]
Fri, 3 Apr 2026 18:53:25 UTC (900 KB)
[v4]
Sun, 11 Oct 2026 21:46:50 UTC (893 KB)
来源：arXiv cs.AI · arxiv.org