arXiv cs.CL· Yuning Bian, Lele Xu, Yifan Yuan, Xinyu Yin, Chengyan Zhan·· 13 小时前AI 评分31
真实文档修订下的分块重计算：测量固定、递归与内容定义分块的更新成本与检索质量
Chunk Recomputation under Real Document Revisions: Measuring Update Cost and Retrieval Quality of Fixed, Recursive, and Content-Defined Chunking
AI 导读
研究在120对中英文维基百科相邻修订上测量三种无状态分块器的更新成本，在116篇共同成功样本中，固定长度分块需重计算新版本嵌入输入的57-68%，递归分块为38-56%，内容定义分块为43-54%（62篇中文文章中有4篇失败）。
正文 · 原文
Abstract:Chunking strategies for retrieval-augmented generation are almost always evaluated on a static snapshot, yet the documents underlying a knowledge base continue to change. Because chunks are embedded independently, a boundary that moves after an edit forces unchanged text to be re-embedded. We measure this update cost on 120 pairs of adjacent Chinese and English Wikipedia revisions for three stateless chunkers (non-overlapping fixed-size, recursive paragraph-first, and content-defined chunking) under a strict byte-identical reuse rule and a frozen dense encoder. On the 116-article common-success pool, with frozen, length-calibrated parameters, fixed-size chunking recomputes 57-68% of its full new-version embedding-input workload, recursive chunking 38-56%, and content-defined chunking 43-54% (it failed on 4 of 62 Chinese articles), while at these parameters mean evidence recall under a fixed 2048-token retrieval budget differs by less than two percentage points within either language. A cost-only parameter grid shows that the ranking of the two content-aware chunkers changes with chunk-length settings, whereas non-overlapping fixed-size chunking has the highest recomputation cost in every length-matched condition. Controlled edits and a verbatim-presence diagnostic support boundary drift as one contributor to recomputation of text already present in the previous version. Update cost is therefore a substantial, parameter-sensitive dimension that static retrieval evaluations do not expose and that should be reported alongside retrieval quality.
| Comments: | 6 pages, 3 figures, 3 tables. Accepted at AIBDCC 2026 |
| Subjects: | Computation and Language (cs.CL); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2610.31019 [cs.CL] |
| (or arXiv:2610.31019v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31019 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuning Bian [view email]
[v1]
Thu, 22 Oct 2026 09:54:17 UTC (109 KB)
来源：arXiv cs.CL · arxiv.org