arXiv cs.AI· Zhanke Zhou, Zongze Li, Weikai Huang, Xuan Li, Chentao Cao, Xiao Feng, Xiangyu Lu, Jinbo Hu, Yi Xie, Nico Pelleriti, Shiyang Liu, Menghan Lu, Max Zimmer, Brando Miranda, Jiangchao Yao, Bo Liu, Sanmi Koyejo, Sebastian Pokutta, Bo Han·· 2 天前AI 评分32
AlphaDiana：面向智能体可验证推理任务的测量系统
AlphaDiana: A Measurement System for Agents in Verifiable Reasoning Tasks
AI 导读
AlphaDiana 是一个模块化测量系统，通过数据集、智能体、环境和评分器的接口，在保留 harness 原生控制循环的同时分别控制文件复用与记忆保留。研究用 3 个开源模型和 3 个开源 harness 在 5 个答案型基准上测试，发现 harness 并不能稳定提升推理准确率：在模型单独答错的问题上取得的增益，可能被其原本答对问题上的损失抵消。
正文 · 原文
Authors:Zhanke Zhou, Zongze Li, Weikai Huang, Xuan Li, Chentao Cao, Xiao Feng, Xiangyu Lu, Jinbo Hu, Yi Xie, Nico Pelleriti, Shiyang Liu, Menghan Lu, Max Zimmer, Brando Miranda, Jiangchao Yao, Bo Liu, Sanmi Koyejo, Sebastian Pokutta, Bo Han
Abstract:Agents pair reasoning models with harnesses that organize model calls, execute tools, load skill instructions, and retain history in memory. Understanding agent performance requires linking execution conditions to outcomes, behavior, and resource use. We introduce AlphaDiana, a measurement system with modular interfaces for datasets, agents, environments, and scorers. It preserves harness-native control loops while controlling file reuse and memory retention separately. AlphaDiana supports macro analyses of complete agents and micro interventions on agent components such as tools, skills, and memory under matched conditions. We study three open-source models and three open-source harnesses on five answer-based benchmarks, with two models also tested on two interactive benchmarks. We find that harnesses do not consistently improve the accuracy of reasoning: gains on questions unsolved by the model alone can be outweighed by losses on questions it answers correctly without tools. This inconsistency extends to component interventions: neither skill guidance nor retained history reliably improves accuracy. Behavior and output-token use must be assessed with correctness: low token entropy can accompany wrong answers, and less output per execution can mean more output tokens per correct answer when accuracy falls. AlphaDiana links answer correctness and execution status to traces and resource records for comparing agents and diagnosing failures under shared task and resource policies.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.20675 [cs.AI] |
| (or arXiv:2610.20675v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20675 arXiv-issued DOI via DataCite |
Submission history
From: Zhanke Zhou [view email]
[v1]
Wed, 14 Oct 2026 14:04:48 UTC (3,312 KB)
来源：arXiv cs.AI · arxiv.org