arXiv cs.AI· Shriram Vasudevan, Videet Mehta, Connor Sweeney·· 1 天前AI 评分38
BABA-Bench：评估长程患者权益倡导智能体
BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents
AI 导读
研究团队推出 BABA-Bench，一个由 47 个交互世界组成的基准，每个世界均基于经专家审核的去标识化真实患者权益倡导案例构建，用于检验 LLM 智能体能否在达成正确结果的同时遵守必要流程。
正文 · 原文
Abstract:An LLM agent that helps a patient get care must do two things. It must reach the correct outcome, and it must follow the necessary procedure. For example, the agent must get the patient's permission before it shares the patient's records. We introduce BABA-Bench, a benchmark of 47 interactive worlds. We built each world from a de-identified, real patient advocacy case that expert advocates reviewed. In each world, the agent must find hidden information and coordinate insurers, clinics, suppliers, pharmacies, and community organizations. The engine checks each action at the time the agent does it. It checks if the agent has permission, if the agent has evidence for the action, and if the action is prohibited. We evaluated 14 frontier models with three runs on each world, for a total of 1,974 runs. Agents often reached the correct outcome by an invalid route. Of the 921 runs that reached all required outcomes, 254 (28%) did not follow the necessary procedure. Claude Opus 5 reached all required outcomes in 81.6% of its runs, but only 52.5% of its runs passed all checks. When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%. For these models, the main problem is to find a route, not to execute it.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.41076 [cs.AI] |
| (or arXiv:2610.41076v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41076 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Videet Mehta [view email]
[v1]
Thu, 29 Oct 2026 23:46:24 UTC (401 KB)
来源：arXiv cs.AI · arxiv.org