arXiv cs.AI· Zipeng Ye, Shenyu Dai, Yong Xie, Kun Wang, Jiyang Guan, Zhikai Chen, Jiaqi Weng, Jialing Tao, Hui Xue·· 1 天前AI 评分32
ReA-Bench：面向负责任 AI 智能体审计的诊断基准
ReA-Bench: Benchmarking Responsible Agent Auditing
AI 导读
研究者推出 ReA-Bench，一个用于审计负责任 AI 智能体行为的诊断基准，将审计挑战分为内部安全知识、轨迹内矛盾、长上下文证据检索、来源与推理验证、间接引导、信息缺失六个层级。该基准包含动作看似有风险但符合任务与授权的良性案例，并提供成对轨迹版本以检验审计者在轨迹缩短或补充缺失信息后能否更可靠地识别同一风险。多数样本取自真实生产轨迹，场景经人工筛选与改编以保留真实任务及执行中可能出现的风险。
正文 · 原文
Abstract:As agents perform tasks autonomously in local and cloud environments, auditing their execution trajectories can help service providers identify risks and improve agent harnesses and foundation models. Our analysis of production and red-team trajectories shows that some risks are repeatedly missed across auditing configurations. Overall recall obscures which risks are missed and whether higher recall comes with more false positives on legitimate behavior. Existing benchmarks offer limited insight into these failures. Generated trajectories can contain inconsistent tool calls and responses, conspicuous risky actions, or repetitive attack patterns. Grouping results by risk type also makes it difficult to tell whether auditors overlooked evidence, accepted an unsupported conclusion, or lacked information needed to make a judgment.
We introduce ReA-Bench, a diagnostic benchmark for auditing responsible agent behavior. It organizes auditing challenges into six tiers: internal safety knowledge, contradictions within trajectories, evidence retrieval in long contexts, verification of sources and reasoning, indirect steering, and missing information. It includes challenging benign cases in which actions resemble risky ones but are consistent with the task and its authorization. It also provides paired trajectory versions to test whether auditors detect the same risks more reliably when the trajectory is shortened or missing information is supplied. Most samples are drawn from real production trajectories, with scenarios manually selected and adapted to preserve realistic tasks and risks that could plausibly arise during execution.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.16493 [cs.AI] |
| (or arXiv:2611.16493v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.16493 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zipeng Ye [view email]
[v1]
Thu, 12 Nov 2026 09:01:24 UTC (2,283 KB)
来源：arXiv cs.AI · arxiv.org