跳到正文
arXiv cs.CL· Jiayu Feng·· 2 天前AI 评分31

GT-FENG:语言模型事实评估与规范依据的通用理论框架

GT-FENG: General Theory of Factual Evaluation and Normative Grounding in Language Models *Author(s)

AI 导读

GT-FENG 是首个统一法律、政策等规则领域上下文/时间正确性与规则化错误归因的通用框架,可对照适用规则检查答案并找出与错误答案匹配的成文规则。在两项模型的 Medicaid 收入上限问答中,记录被问州的自身规则后,仅匹配其他州数值的错误答案减少 68-78%,且过半稳定错误答案随后匹配两个及以上成文来源。该框架以开源 Python 包 gtfeng 发布。

正文 · 原文

View PDF HTML (experimental)

Abstract:Whether a language model's answer is correct often depends on context: the same value can be right under one jurisdiction, date, or legal definition and wrong under another. We introduce GT-FENG (General Theory of Factual Evaluation and Normative Grounding), the first general framework to unify contextual/temporal correctness and rule-based error attribution across law, policy, and other rule-governed domains. GT-FENG checks answers against the applicable rules and identifies documented rules that match incorrect answers. It shows that when multiple rules give the same value, the answer alone cannot tell which rule the model used. In Medicaid income-limit answers from two models, documenting the asked state's own records reduces the number of wrong answers that match only other states' values by 68-78%, and more than half of stable wrong answers then match two or more documented sources. GT-FENG provides a foundation for evaluating language models in rule-governed domains and for future benchmarks and evaluation toolkits. We release GT-FENG as an open-source Python package, gtfeng.
Comments: 7 pages, 4 tables. Code: this https URL
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2611.18279 [cs.CL]
  (or arXiv:2611.18279v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.18279

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Jiayu Feng [view email]
[v1] Fri, 13 Nov 2026 15:11:11 UTC (28 KB)

来源:arXiv cs.CL · arxiv.org