arXiv cs.CL· Jiayi Kuang, Yinghui Li, Yunze Song, Keyu Chen, Zhifeng Shen, Yangning Li, Yidong Wang, Di Yin, Ruizhi Qiao, Xing Sun, Kai Jin, Ying Shen, Liang Lin, Philip S. Yu·· 13 小时前AI 评分31
从原子能力到智能体：面向 LLM 智能体数学能力的可解释评测框架
From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilities
AI 导读
研究者提出一个过程级基准，用于评测 LLM 的智能体数学推理能力，将解题行为对齐到可复用数学原子能力的结构化分类体系。该框架覆盖文本与多模态场景下的规划、行动和反馈任务，并通过自动化流水线合成高质量轨迹、借助受控 LLM 改写生成细粒度标注。实验显示，端到端准确率相近的模型在智能体能力画像上存在明显差异。
正文 · 原文
Authors:Jiayi Kuang, Yinghui Li, Yunze Song, Keyu Chen, Zhifeng Shen, Yangning Li, Yidong Wang, Di Yin, Ruizhi Qiao, Xing Sun, Kai Jin, Ying Shen, Liang Lin, Philip S. Yu
Abstract:Large Language Models (LLMs) are evolving from performing end-to-end mathematical reasoning to integrating agentic intelligence. However, most existing math benchmarks evaluate only final answers. This outcome-oriented evaluation provides limited diagnostic value for identifying process-level failures or rigorous logic, failing to guide the transformation of LLMs into robust agents. To bridge this gap, we present a process-level benchmark designed to evaluate the inherent agentic mathematical reasoning abilities of LLMs. Our framework aligns problem-solving agentic behaviors with a structured taxonomy of reusable mathematical atomic capabilities. We design a comprehensive suite of planning, action, and feedback tasks across both textual and multimodal contexts, supported by an automated pipeline that synthesizes high-quality trajectories and produces fine-grained annotations via controlled LLM rewriting. Experiments reveal that models with similar end-to-end accuracy can exhibit markedly different agentic capability profiles. This demonstrates that process-level evaluation is crucial for interpreting the true potential of LLMs and guiding the development of next-generation mathematical agents.
| Comments: | EMNLP 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2608.26950 [cs.AI] |
| (or arXiv:2608.26950v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.26950 arXiv-issued DOI via DataCite |
Submission history
From: Yinghui Li [view email]
[v1]
Thu, 27 Aug 2026 10:52:06 UTC (1,135 KB)
[v2]
Wed, 21 Oct 2026 01:46:52 UTC (1,142 KB)
来源：arXiv cs.CL · arxiv.org