arXiv cs.AI· Zidi Xiu, David Q. Sun, Kevin Cheng, Maitrik Patel, Josh Date, Yizhe Zhang, Jiarui Lu, Omar Attia, Raviteja Vemulapalli, Oncel Tuzel, Meng Cao, Samy Bengio·· 12 小时前AI 评分38
ASTRA-bench：评估工具使用智能体的个性化上下文推理与行动规划
ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context
AI 导读
研究者发布 ASTRA-bench，一个将时间演变的个人上下文、交互式工具箱与复杂用户意图统一起来的工具使用智能体基准，通过事件驱动流水线生成覆盖四个主角的 2,413 个场景。对 Claude-4.5-Opus、DeepSeek-V3.2 等前沿模型的评测显示，高复杂度条件下性能显著下降，参数生成成为主要瓶颈。基准已随完整执行环境与评估脚本开源发布。
正文 · 原文
Authors:Zidi Xiu, David Q. Sun, Kevin Cheng, Maitrik Patel, Josh Date, Yizhe Zhang, Jiarui Lu, Omar Attia, Raviteja Vemulapalli, Oncel Tuzel, Meng Cao, Samy Bengio
Abstract:Next-generation AI must manage vast personal data, diverse tools, and multi-step reasoning, yet most benchmarks remain context-free and single-turn. We present ASTRA-bench (Assistant Skills in Tool-use, Reasoning \& Action-planning), a benchmark that uniquely unifies time-evolving personal context with an interactive toolbox and complex user intents. Our event-driven pipeline generates 2,413 scenarios across four protagonists, grounded in longitudinal life events and annotated by referential, functional, and informational complexity. Evaluation of state-of-the-art models (e.g., Claude-4.5-Opus, DeepSeek-V3.2) reveals significant performance degradation under high-complexity conditions, with argument generation emerging as the primary bottleneck. These findings expose critical limitations in current agents' ability to ground reasoning within messy personal context and orchestrate reliable multi-step plans. We release ASTRA-bench with a full execution environment and evaluation scripts to provide a diagnostic testbed for developing truly context-aware AI assistants.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2603.01357 [cs.AI] |
| (or arXiv:2603.01357v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2603.01357 arXiv-issued DOI via DataCite |
Submission history
From: Zidi Xiu [view email]
[v1]
Mon, 2 Mar 2026 01:34:48 UTC (5,955 KB)
[v2]
Thu, 5 Nov 2026 06:20:21 UTC (5,955 KB)
来源：arXiv cs.AI · arxiv.org