arXiv cs.CL· Ruotian Peng, Yuanyi Ren, Shuaiqi Liu, Xin Guo, Fang Zhu, Yangyi Huang, Yue Wu, Zheng Wang, Yandong Wen, Zhihang Fu·· 12 小时前AI 评分49
VibeTraining-Bench：面向迭代系统改进的数据-训练器-推理框架协同设计基准
VibeTraining-Bench: Data-Trainer-Harness Co-Design for Iterative System Improvement
AI 导读
VibeTraining-Bench 是评估研究型智能体跨组件协同设计能力的基准，覆盖终端操作、软件工程、多轮工具调用与数学推理任务。在 SWE-bench Verified 上，GPT-6-Astra 将目标智能体系统性能从 32.9% 提升至 53.7%，接近人工开发的 instruct 模型 55.4%。
正文 · 原文
Abstract:Recursive self-improvement (RSI) requires agents to diagnose system bottlenecks and coordinate improvements across training and inference. We refer to the agents' capability to jointly optimize training data, the trainer, and the inference harness as cross-component co-design. We introduce VibeTraining-Bench, a benchmark for evaluating this capability across tasks spanning terminal operation, software engineering, multi-turn tool use, and mathematical reasoning. Within each task, research agents start from an identical initial agent system under a fixed resource budget. Guided by validation feedback, they iteratively refine data, trainers, and inference harnesses, then select and submit a final agent system for evaluation on the test set. To prevent reward hacking, we enforce three safeguards: test data isolation, separate research and grading containers, and network restrictions for research agents. Our evaluation results show that frontier research agents substantially improve target agent systems. For example, on SWE-bench Verified, GPT-6-Astra raises the target agent system's performance from 32.9% to 53.7%, comparable to the human-developed instruct model's 55.4%. However, there is still room to improve their ability to co-design and iteratively refine data, trainer, and harness to achieve cumulative performance gains. We hope that VibeTraining-Bench will serve as a unified platform for advancing RSI, deepening our understanding of cross-component optimization for agent systems. Code and data are available at this https URL.
| Comments: | (Technical report (27 pages, 21 figures) |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.26342 [cs.AI] |
| (or arXiv:2610.26342v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.26342 arXiv-issued DOI via DataCite |
Submission history
From: Ruotian Peng [view email]
[v1]
Mon, 19 Oct 2026 13:10:54 UTC (1,068 KB)
[v2]
Tue, 20 Oct 2026 13:03:07 UTC (1,067 KB)
来源：arXiv cs.CL · arxiv.org