arXiv cs.AI· Zenghui Zhou, Xiaoke Fang, Zheng Zheng·· 12 小时前AI 评分38
LGMT：一款测试 LLM 逻辑推理可靠性的开源工具
LGMT: A Tool for Testing the Logical Reasoning Reliability of LLMs
AI 导读
LGMT 是一款开源工具，通过基于一阶逻辑的蜕变关系检测 LLM 的推理不一致性：给定前提-结论形式的源测试用例，它生成逻辑等价的后续用例并检查模型蕴含预测是否一致，无需真实标签。
正文 · 原文
Abstract:Large Language Models (LLMs) increasingly serve as reasoning components in modern software systems, yet a model that answers a logical query correctly can contradict itself when the same problem is restated in an equivalent form. Static benchmarks evaluate queries in isolation and therefore fail to expose such inconsistencies. To address this limitation, we present LGMT, an open-source tool that detects reasoning inconsistencies through metamorphic relations grounded in first-order logic. Given a source test case in premise-conclusion form, LGMT generates logically equivalent follow-up cases and checks whether the model's entailment predictions agree across the pair, without ground-truth labels. The tool provides 20 built-in metamorphic relations, supports custom ones verified by the Z3 theorem prover, and runs from a CLI or a web studio, including an offline mock mode requiring no API key. Across six widely used LLMs on a bundled benchmark of 3,091 metamorphic groups, LGMT reveals inconsistency rates ranging from 20.84% to 29.44%. LGMT is publicly available on GitHub at this https URL, with a demonstration video at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32627 [cs.AI] |
| (or arXiv:2610.32627v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32627 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zenghui Zhou [view email]
[v1]
Fri, 23 Oct 2026 13:41:43 UTC (766 KB)
来源：arXiv cs.AI · arxiv.org