跳到正文
arXiv cs.CL· Lan Zhang, Marco Valentino, Jordan Meadows, Andr\'e Freitas·· 2 天前AI 评分24

超越黄金标准:面向形式数学推理的 LLM 评审员认知集成框架

Beyond Gold Standards: Epistemic Ensemble of LLM Judges for Formal Mathematical Reasoning

AI 导读

研究提出 EFG(认知与形式化基础集成)LLM 评审员集成方法,用于自动评估自然语言陈述到形式语言的自动形式化任务,评判标准涵盖逻辑保持(LP)、数学一致性(MC)、形式质量(FQ)与形式有效性(FV)。

正文 · 原文

View PDF HTML (experimental)

Abstract:Statement autoformalization plays a crucial role in formal mathematical reasoning by enabling the automatic translation of natural language statements into formal languages. While recent advances using large language models (LLMs) have shown promising capabilities in autoformalization, methods for automatically evaluating autoformalization remain underexplored. LLM-as-a-judge presents a promising approach for automating such evaluation, however, existing methods typically employ coarse-grained and generic evaluation criteria, which limit their effectiveness for advanced formal mathematical reasoning, where quality hinges on nuanced, multi-granular dimensions. In this work, we take a step toward addressing this gap by introducing a systematic, automatic method to evaluate autoformalization tasks. The proposed method is based on an epistemically and formally grounded ensemble (EFG) of LLM judges, defined on criteria encompassing logical preservation (LP), mathematical consistency (MC), formal quality (FQ), and formal validity (FV), resulting in a transparent assessment that accounts for different contributing factors. We validate the proposed framework to serve as a proxy for autoformalization assessment within the domain of formal mathematics. Overall, our experiments demonstrate that the EFG ensemble of LLM judges is a more suitable emerging proxy for evaluation than a coarse-grained model. These findings suggest that LLM judges, especially when guided by a well-defined set of atomic properties, could offer a scalable, interpretable, and reliable support for evaluating formal mathematical reasoning.
Comments: Accepted to EACL 2027
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2506.10903 [cs.CL]
  (or arXiv:2506.10903v3 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2506.10903

arXiv-issued DOI via DataCite

Submission history

From: Lan Zhang [view email]
[v1] Thu, 12 Jun 2025 17:09:51 UTC (328 KB)
[v2] Fri, 21 Aug 2026 11:36:28 UTC (165 KB)
[v3] Fri, 13 Nov 2026 07:48:18 UTC (162 KB)

来源:arXiv cs.CL · arxiv.org