arXiv cs.CL· Kanghyeon Kim, Johannes F. Lutzeyer, Yanzhu Guo·· 2 天前AI 评分31
注意力图曲率用于数学推理验证
Attention-Graph Curvature for the Verification of Mathematical Reasoning
AI 导读
研究提出将语言模型注意力图的有向 Forman 曲率作为逐步验证数学推理正确性的信号，在 ProcessBench 上为四个逐步验证器追加曲率输入后，全部验证器的平均性能均有提升，其中以全部 token 为输入的 ReProbe 提升最大。代价是模型前向传播时间平均增加 18%。代码已公开。
正文 · 原文
Abstract:Language models have recently solved open Erdős problems and achieved gold-medal level performance on International Mathematical Olympiad problems. But their solutions, written in natural language, cannot yet be verified by proof assistants. An erroneous reasoning step can invalidate every step that comes after, so it is advantageous to detect errors early. Several methods have been proposed to detect erroneous reasoning quickly based on the features of the forward pass of the language model that performs reasoning. In this work, we study whether the directed Forman curvature of that model's attention graph, a local quantity computed at an edge from the attention weights around it, carries information about a step's correctness. To do so, we evaluate four per-step verifiers on ProcessBench with and without the curvature appended to their input. Appending curvature values improves the overall mean performance of all four verifiers. ReProbe, the verifier that takes every token of a step as input, shows the largest improvement. Adding the curvature increases the time of the language model's forward pass by 18% on average. Our code is available at this https URL.
| Comments: | Accepted at the 6th Workshop on Mathematical Reasoning and AI (MATH-AI), NeurIPS 2026. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.16709 [cs.LG] |
| (or arXiv:2610.16709v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16709 arXiv-issued DOI via DataCite |
Submission history
From: Kanghyeon Kim [view email]
[v1]
Mon, 12 Oct 2026 12:43:47 UTC (30 KB)
[v2]
Mon, 26 Oct 2026 23:27:18 UTC (30 KB)
来源：arXiv cs.CL · arxiv.org