arXiv cs.AI· Wei Chen, Peilun Zhou, Zhaoyu Hu, Jiajun Chai, Derong Xu, Zhongni Hou, Yufei Zhang, Guojun Yin, Wei Lin, Zhi Zheng, Tong Xu·· 14 小时前AI 评分40
ATLAS：美团小团工具调用智能体的双时间尺度诊断评估框架
ATLAS: Dual-Horizon Diagnostic Evaluation for Tool-Use Agents in Meituan Xiaotua
AI 导读
美团提出 ATLAS 双时间尺度诊断评估框架，用于工业级工具调用智能体：轮内评估沿思考、反思、工具与技能执行、回复生成追踪执行失败，跨轮评估则诊断多生命周期中用户意图、约束与纠正的维护问题。该框架已在百万级 DAU 的本地生活助手美团小团落地，14 天在线 A/B 测试（每组近 200 万用户）显示答案阅读率提升 2.84%、支付订单量提升 6.13%，并已全量上线。
正文 · 原文
Authors:Wei Chen, Peilun Zhou, Zhaoyu Hu, Jiajun Chai, Derong Xu, Zhongni Hou, Yufei Zhang, Guojun Yin, Wei Lin, Zhi Zheng, Tong Xu
Abstract:LLM-based assistants on web-scale service platforms increasingly address local-service needs through multi-step planning and tool use. Under dynamic business conditions and evolving user needs, outcome-level evaluation offers limited evidence of where execution fails or how service drifts across continued interaction. We present ATLAS, a dual-horizon diagnostic evaluation framework for industrial tool-use agents. Within-turn evaluation uses trajectory-wise signals across thinking and reflection, tool and skill execution, and response generation to identify execution failures and unmet capability requirements. Across-turn evaluation uses user-wise signals spanning multiple lifecycles to diagnose failures in maintaining or updating user intent, constraints, and corrections. Fine-grained signals use rule-based checks and LLM judgments with explicit evidence scopes and decision boundaries. To support reliable diagnosis at production scale, LLM judges are calibrated against dynamically maintained references from business logs, and selected judges are distilled into compact diagnostic models. The resulting diagnostic profiles help developers attribute failures, prioritize targeted interventions, and reassess improvements. Applied in Meituan Xiaotuan, a million-scale DAU local-services assistant, ATLAS achieves high diagnostic fidelity across calibrated judges and distilled models. Blinded human auditing and a 14-day online A/B test with nearly two million users per arm confirm that the ATLAS-guided agent improves recommendation relevance, factual reliability, user engagement (+2.84% answer read-through rate), and order volume (+6.13% paid orders), leading to full production deployment.
| Comments: | 19 pages; The project's homepage is this https URL |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.30685 [cs.AI] |
| (or arXiv:2608.30685v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.30685 arXiv-issued DOI via DataCite |
Submission history
From: Wei Chen [view email]
[v1]
Mon, 31 Aug 2026 12:25:24 UTC (7,470 KB)
[v2]
Thu, 29 Oct 2026 08:57:19 UTC (1,961 KB)
来源：arXiv cs.AI · arxiv.org