arXiv cs.AI· Shaikh Akib Shahriyar, MD. Akil Raihan Iftee, Armon Barton, Matthew Wright·· 3 天前AI 评分29
什么才算同一个任务？面向工具型智能体鲁棒性评估的可执行任务契约
What Counts as the Same Task? Executable Contracts for Robustness Evaluation of Tool-Using Agents
AI 导读
针对工具型智能体鲁棒性测试中"改动后的用例是否仍属同一任务"判定薄弱的问题，研究者提出可执行任务契约，在智能体运行前声明任务定义事实、允许的编辑和完成标准并逐例校验。
正文 · 原文
Abstract:Robustness tests for tool-using agents often change the agent's environment, for example by hiding a hostile note in an invoice, and then judge the agent by what it does. The result tells us about the original task only if the changed test case is still a valid test of that task. Many evaluations decide this with weak rules, such as accepting any well-formed edit or any edit that stays close to the original in embedding space. We propose executable task contracts. A contract states which facts define a task, which edits are allowed, and what finishing the task means, and it checks each changed test case before the agent runs. We compare validity rules by scoring the same 14,336 stored agent runs under each rule. Across four encoders, embedding thresholds allowed 75-100% of edits that changed the task, such as swapping the payee, yet rejected 24-45% of valid hostile notes. With a MiniLM cosine threshold, Mistral appeared vulnerable on all 64 tasks, but only 26 had a successful valid attack. Replaying a saved solution and allowing edits to benchmark fields fail in similar ways. Evaluations should state, and check, what counts as the same task.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2610.14477 [cs.SE] |
| (or arXiv:2610.14477v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14477 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shaikh Akib Shahriyar [view email]
[v1]
Sat, 10 Oct 2026 02:18:56 UTC (451 KB)
来源：arXiv cs.AI · arxiv.org