arXiv cs.CL· Zihan Chen, Di Zhu, Lei Nico Zheng·· 12 小时前AI 评分36
递归自我改进（RSI）声明的评估证据审计：114 篇研究全文编码显示产品评估多于过程验证
Auditing the Product, Not the Recursion: The Evaluation Evidence Behind Recursive Self-Improvement Claims
AI 导读
一项对 2026 年 10 月冻结的 arXiv 检索框架内 114 篇合格研究的全文编码发现，90 篇自称 RSI 的研究中 90% 报告了超出开发任务的产品评估，但仅 22% 审计了完整循环的两次以上重跑。在声称改进流程更优的 26 篇研究中，23 篇给出相关比较，仅 4 篇在相同起始产物与匹配预算下比较了前后流程。结果支持按声明索引的报告标准，将每项声明与其所需的评估设计对应。
正文 · 原文
Abstract:Recursive self-improvement (RSI) promises AI systems, such as self-evolving LLM agents, that become better at producing further improvements. Evaluating this promise requires separating product claims about the final system from process claims about the improvement process itself. Existing surveys specify the evidence these claims require, but how often studies report that evidence remains unmeasured. We examine this question through full-text coding of 114 eligible studies in a reproducible arXiv search frame frozen in October 2026, including 90 that describe their own systems as RSI. The codebook links claims to evaluation designs. Independent human coding, third-reader adjudication, and a separate blind coding of improver comparisons assess coding reliability. Among the 90 RSI studies, 90% report product evaluation beyond development tasks under a broad criterion that includes partial or unstated separation, while 60% report evaluation on tasks unused for selection. Only 22% audit two or more reruns of the full loop. In a human-coded random sample, the RSI studies show the same contrast: 45% evaluate on tasks unused for selection and 14% audit two or more reruns. Of the 26 studies claiming a better improvement procedure, 23 report a related comparison, but only 4 compare earlier and later procedures from identical starting artifacts at a matched budget. Across these results and a comparison set of established works, evidence for repeatability and improved procedures remains less developed than product evaluation. Three propositions explain how selection effects, shared evaluator errors, and changes in system state limit what product evaluations can establish. Together, the findings support a claim-indexed reporting standard that connects each claim to the evaluation design needed to test it.
| Comments: | 39 pages, 1 figure, 20 tables |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE) |
| ACM classes: | I.2.6; I.2.11; G.3 |
| Cite as: | arXiv:2610.27610 [cs.LG] |
| (or arXiv:2610.27610v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.27610 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zihan Chen [view email]
[v1]
Tue, 20 Oct 2026 04:23:05 UTC (145 KB)
来源：arXiv cs.CL · arxiv.org