arXiv cs.AI· Jiayu Ding, Shuming Ma, Lei Cui, Nanning Zheng, Furu Wei·· 1 天前AI 评分31
LongReasonArena：面向大语言模型的长推理基准
LongReasonArena: A Long Reasoning Benchmark for Large Language Models
AI 导读
研究者提出 LongReasonArena，一个专门评估 LLM 长推理能力的基准，任务要求模型执行多步算法（含检索与回溯），推理长度可任意扩展至 100 万 token。评测显示开源与闭源模型均受显著挑战，GPT-5 在 Level 3 上准确率仅 27.8%，且准确率随预期推理步数对数呈线性下降。代码与数据已公开。
正文 · 原文
Abstract:Existing long-context benchmarks for Large Language Models (LLMs) focus on evaluating comprehension of long inputs, while overlooking the evaluation of long reasoning abilities. To address this gap, we introduce LongReasonArena, a benchmark specifically designed to assess the long reasoning capabilities of LLMs. Our tasks require models to solve problems by executing multi-step algorithms that reflect key aspects of long reasoning, such as retrieval and backtracking. By controlling the inputs, the required reasoning length can be arbitrarily scaled, reaching up to 1 million tokens of reasoning for the most challenging tasks. Extensive evaluation results demonstrate that LongReasonArena presents a significant challenge for both open-source and proprietary LLMs. For instance, even GPT-5 achieves only 27.8% accuracy on Level 3. Further analysis also reveals that the accuracy exhibits a linear decline with respect to the logarithm of the expected number of reasoning steps. Our code and data are available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2508.19363 [cs.CL] |
| (or arXiv:2508.19363v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2508.19363 arXiv-issued DOI via DataCite |
|
| Journal reference: | Findings of the Association for Computational Linguistics: EMNLP 2026, pp. 2470-2485, 2026 |
| Related DOI: | https://doi.org/10.18653/2026.findings-emnlp.120
DOI(s) linking to related resources |
Submission history
From: Jiayu Ding [view email]
[v1]
Tue, 26 Aug 2025 18:41:53 UTC (364 KB)
[v2]
Wed, 11 Nov 2026 09:42:33 UTC (376 KB)
来源：arXiv cs.AI · arxiv.org