arXiv cs.CL· Samuel Xiao, Judy Song, Rory Hu, Ziliang Zong·· 10 小时前AI 评分31
NL2AGBench:评估 LLM 将几何题自动形式化为 AlphaGeometry 输入的基准
NL2AGBench: Benchmarking LLM Auto-Formalization for AlphaGeometry
AI 导读
研究者提出 NL2AGBench,用 AlphaGeometry 内部执行验证来评估 LLM 把英文几何题转成其专用 DSL 的能力,而非仅比对文本相似度。在十个开源与闭源 LLM 上,领先闭源模型可执行翻译率超过 80%,最大的开源模型仍难以稳定保留几何约束并生成合法形式化。研究还给出区分语法与逻辑错误的错误分类体系,并测试 few-shot 提示、微调与人工提示等改进策略。
正文 · 原文
Abstract:Recent advances in large language models (LLMs) have demonstrated strong capabilities in natural language understanding and mathematical reasoning. However, their ability to translate informal mathematical problems into formal representations remains underexplored. This limitation is particularly important for neuro-symbolic geometry systems such as AlphaGeometry, whose theorem-proving engine requires inputs in a specialized domain-specific language (DSL). Although AlphaGeometry achieves near-IMO gold-medalist performance, manually converting natural-language problems into its formal syntax remains a significant usability bottleneck. To address this challenge, we introduce the Natural Language to AlphaGeometry Benchmark (NL2AGBench), which evaluates LLMs in translating English geometry problems into AlphaGeometry-compatible formal representations. NL2AGBench uses execution-based verification within AlphaGeometry to assess translation quality rather than relying solely on textual similarity. We evaluate ten state-of-the-art open- and closed-source LLMs across multiple parameter scales and analyze executable translation accuracy, syntactic correctness, and error characteristics. Our experiments reveal a substantial performance gap between closed- and open-source models: leading closed-source models achieve executable translation rates above 80%, while even the largest open-source models struggle to consistently preserve geometric constraints and produce valid formalizations. We introduce an error taxonomy distinguishing syntax and logic errors and investigate mitigation strategies, including few-shot prompting, fine-tuning, and human-guided hinting, which yield measurable improvements across multiple model families.
| Comments: | Published at the 2026 IEEE International Conference on Foundation and Large Language Models (FLLM 2026) |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.28481 [cs.CL] |
| (or arXiv:2608.28481v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.28481 arXiv-issued DOI via DataCite |
Submission history
From: Ziliang Zong [view email]
[v1]
Fri, 28 Aug 2026 16:07:16 UTC (1,180 KB)
[v2]
Wed, 18 Nov 2026 21:47:25 UTC (1,846 KB)
来源:arXiv cs.CL · arxiv.org