arXiv cs.CL· Shuofei Qiao, Xuanguang Pan, Yijun Chen, Wendi Feng, Bin Wu, Runnan Fang, Yuqi Zhu, Shumin Deng, Emine Yilmaz·· 12 小时前AI 评分38
Δτ-Bench：推动智能体适应不断变化的能力边界
$\Delta\tau$-Bench: Pushing Agents Toward a Moving Capability
AI 导读
研究团队提出 Δτ-Bench，一个模拟大型工具生态结构演化的可执行智能体基准，包含 42 个领域、1,588 个多轮任务和六个连续生态状态。评测显示最强模型或框架任务成功率仅 42.3%，现有技能学习方法带来的初始收益无法随生态演化持续转化为适应能力。为此提出生态管理器 Steward，通过增量技能维护与任务执行中的动态供给，在生态变化下保持更高成功率并降低技能维护开销。
正文
Abstract:In this paper, we introduce $\Delta\tau$-Bench, an executable agent benchmark that models the structural evolution of large tool ecosystems. In $\Delta\tau$-Bench, independently evolving services continually reshape how tasks can be accomplished: familiar tools may disappear or be replaced, while more efficient alternatives emerge amid overlapping functionality and noisy documentation. Against this moving capability frontier, we evaluate whether agents can proactively discover efficient tool-use paths and promptly revise their accumulated procedural knowledge, bringing together demands that existing benchmarks fail to examine. $\Delta\tau$-Bench comprises 1,588 validated multi-turn tasks across 42 domains and six successive ecosystem states generated through capability-preserving transformations. Our evaluation shows that even the strongest evaluated model or harness achieves only 42.3% task success, while initial gains from existing skill-learning methods can not consistently translate into better adaptation as the ecosystem evolves. These findings motivate Steward, an ecosystem manager that jointly manages tools and procedural knowledge through incremental skill maintenance and dynamic provisioning during task execution. Experiments show that Steward sustains higher task success across ecosystem changes with lower skill-maintenance overhead, while improving both recovery from disrupted workflows and adoption of more efficient alternatives.
| Comments: | Work in progress |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Information Retrieval (cs.IR); Machine Learning (cs.LG); Software Engineering (cs.SE) |
| Cite as: | arXiv:2610.34238 [cs.CL] |
| (or arXiv:2610.34238v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34238 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shuofei Qiao [view email]
[v1]
Sun, 25 Oct 2026 12:10:52 UTC (2,316 KB)
来源：arXiv cs.CL · arxiv.org