arXiv cs.AI· Nishal Thomas, Noel Thomas·· 12 小时前AI 评分34
FormInv：数学推理基准语义不变性测量协议，Claude Haiku 4.5 答案翻转率高达 41.9%
FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks
AI 导读
FormInv 提出 CIR（条件不一致率）与 SCR 两项语义不变性指标，在 9 个模型、366-811 道 Lean4 验证题目上评测。在全部模型均答对的 43 道定理上，Claude Haiku 4.5 的答案翻转率为 41.9%，DeepSeek V3 仅 4.7%。
正文 · 原文
Abstract:A paraphrase-quality audit of MathCheck (ICLR 2025) detected 4 semantically incorrect paraphrases in 129 groups (3.1%); removing them drops GPT-4o from rank 2 to rank 4 and elevates Claude Haiku and DeepSeek V3 above it. Cross-model unanimity found these errors automatically for under $10; in our own dataset the same protocol found that 47% of auto-generated connective-variation paraphrases were semantically incorrect. That flaw compounds a deeper measurement gap. On the theorems every model answers correctly in canonical form, a semantically equivalent rewording still flips the answer for 5-42% of them: Claude Haiku 4.5 flips a known-correct answer about 9x as often as DeepSeek V3 (41.9% vs. 4.7% on the 43 theorems all models share), a difficulty-controlled gap we call the Conditional Inconsistency Rate (CIR). The CIR ranking replicates on an independent 103-theorem set. Raw Semantic Consistency Rates (SCR) span 50-82% across 9 models, but this spread is mostly a mechanical consequence of compounding accuracy over k ~ 6 paraphrases; what CIR isolates is the within-theorem clustering of failures. Because no model dominates on all paraphrase families, the families a benchmark includes implicitly fix its model ranking (a No-Free-Benchmark corollary). FormInv supplies the audit protocol, CIR and SCR as invariance measures, evaluated on 9 models across 366-811 items whose canonical statements are Lean4-verified, and FormInvSelector for regime-aware model selection.
| Comments: | 19 pages, 1 figure. Accepted as a poster at the 3rd AI for Math Workshop (AI4Math), ICML 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.29001 [cs.LG] |
| (or arXiv:2605.29001v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.29001 arXiv-issued DOI via DataCite |
Submission history
From: Noel Thomas [view email]
[v1]
Wed, 27 May 2026 18:59:18 UTC (84 KB)
[v2]
Fri, 16 Oct 2026 12:02:50 UTC (69 KB)
来源：arXiv cs.AI · arxiv.org