arXiv cs.AI· Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo, Ying Zhao, Youmin Chen, Xiaojian Liao, Congming Gao, Tianyang Jiang, Yixiao Chen, Qing Wang, Zhiguo Yu, Xiaqing Li, Yuhao Zhang, Jiawen Sun, Renzhe Xu, De Mi, Tiancheng Wu, Xianya Fu, Xuanlu Zhao, Bo Feng, Meixuan Ren, Guanming Lin, Ziheng Zhou, Xinhua Wang·· 2 天前AI 评分44
FDE-Gym：AI 交付的最后一公里基准
FDE-Gym: The Last Mile of AI Delivery
AI 导读
研究团队推出 FDE-Gym，一个包含 95 个端到端交付任务的基准，覆盖 20 个客户场景、18 个行业，由行业专家从真实交付项目改编。在模拟客户环境中，13 个模型配置（来自 7 家提供商）表现最佳者平均得分仅 0.56，95 个任务中只解决 14 个。发现未明说的约束是主要瓶颈，即便最佳配置已获取全部关键事实的 23 个任务中，也仅 10 个达到 0.8 以上业务提升。
正文 · 原文
Authors:Dongxu Yang, Hongda Yan, Renxin Wang, Baoli Zhang, Haojia Zuo, Ying Zhao, Youmin Chen, Xiaojian Liao, Congming Gao, Tianyang Jiang, Yixiao Chen, Qing Wang, Zhiguo Yu, Xiaqing Li, Yuhao Zhang, Jiawen Sun, Renzhe Xu, De Mi, Tiancheng Wu, Xianya Fu, Xuanlu Zhao, Bo Feng, Meixuan Ren, Guanming Lin, Ziheng Zhou, Xinhua Wang
Abstract:Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world customer delivery is different: forward-deployed engineers (FDEs) must clarify ambiguous requirements, work in live systems, and uncover unstated constraints. We introduce FDE-Gym, a benchmark of 95 end-to-end delivery tasks from 20 customer scenarios across 18 sectors, each adapted by industry experts from real-world delivery projects. In simulated customer environments with scripted stakeholders and modifiable customer systems, agents interact with stakeholders and operate on production-like systems to deliver working solutions. FDE-Gym scores a delivery by how much it improves the customer's own success metric, normalized between 0 (unchanged system) and 1 (reference solution), with penalties for stakeholder interactions (messages, trial runs, and waiting time). An online platform lets humans practice FDE skills under the same grading criteria. We evaluate 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean FDE-Gym score of 0.56 and resolves only 14 of the 95 tasks (a score of at least 0.8 with no incident). Discovering unstated constraints is the primary bottleneck: performance drops sharply when models must uncover them without explicit guidance. Yet information discovery alone does not account for the gap: even on the 23 tasks where the best configuration obtains every key fact from stakeholders, it reaches a normalized business uplift of 0.8 or above on only 10. These results reveal a substantial gap between solving well-specified tasks and delivering effective AI systems for real customers.
| Comments: | 14 pages, 8 figures, 4 tables. Website: this https URL ; Code: this https URL ; Data: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE) |
| Cite as: | arXiv:2610.22350 [cs.AI] |
| (or arXiv:2610.22350v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22350 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dongxu Yang [view email]
[v1]
Thu, 15 Oct 2026 17:57:23 UTC (5,295 KB)
来源：arXiv cs.AI · arxiv.org