arXiv cs.CL· Yuting Ning, Ruijia Cheng, Sushobhan Ghosh, Peitong Duan, HungShi Lin, Abhishek Sundararajan, Shreyash Pandey, Huan Sun, Jeffrey Nichols, Eldon Schoop·· 15 小时前AI 评分42
iPrefBench：面向个性化移动使用智能体的用户中心基准
iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents
AI 导读
iPrefBench 是评估个人移动使用智能体“推断与询问”能力的基准，在可执行 iOS 环境中运行，基于含 3,051 条历史活动事件的连贯人设，覆盖 26 个应用的 130 个欠指定任务。7 个前沿智能体的动作完成分最高达 91.6%，但完整任务成功率最高仅 47.7%，个性化明显落后于执行能力，尤其在只能向用户询问的信息上请求不足。
正文 · 原文
Abstract:An agent that truly knows and acts on behalf of its user has been a long-standing promise of AI assistants. Computer-use agents bring this promise within reach by capably executing fully specified tasks across apps. However, real-world user requests are often underspecified. To act correctly for the particular user, an agent must infer some missing information from the user's activity history and ask for other information that only the user can provide, without making unsupported assumptions or burdening the user with unnecessary questions. We present iPrefBench, a benchmark for evaluating the infer and ask capabilities of personal mobile-use agents in an executable iOS environment with an interactive simulated user. Grounded in a coherent persona with 3,051 past activity events, iPrefBench comprises 130 underspecified user tasks across 26 apps. Each omitted user-dependent requirement is, by construction, either history-inferable or ask-only. iPrefBench evaluates personalized mobile agents on both whether agents complete tasks correctly for the user (i.e., personalized task completion) and whether they seek user input when necessary (i.e., user-agent interaction). Evaluation of 7 frontier agents reveals a substantial gap between execution and personalization. Agents achieve high action completion scores (up to 91.6%) but low full task success (up to 47.7%) because personalization lags behind. They especially struggle with ask-only information, failing to request enough necessary information. Our findings highlight a challenge for personalized agents in knowing what to infer and when to ask. We hope that iPrefBench can serve as a valuable testbed for building agents that know their user and work with them.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.34735 [cs.CL] |
| (or arXiv:2610.34735v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34735 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuting Ning [view email]
[v1]
Mon, 26 Oct 2026 02:35:41 UTC (15,332 KB)
来源：arXiv cs.CL · arxiv.org