arXiv cs.AI· Ernie Hobbs, Jim Hinson·· 4 小时前AI 评分39
GauntletScore:可审计的事实核查引擎在 LLM-AggreFact 上追平最佳已发表核查器
GauntletScore: Auditable Fact-Checking That Matches the Best Published Checkers on LLM-AggreFact
AI 导读
GauntletScore 是一个可审计的事实核查引擎,在 LLM-AggreFact 完整测试集(11 个数据集、29,320 条 claim)上取得 79.51 平衡准确率,高于截至 2026 年 11 月 15 日排行榜上所有已公布平均值,与最佳的 nimble-v4-max(79.45)持平。
正文 · 原文
Abstract:Checking whether a generated sentence is supported by its source document is a standard safeguard for AI-written text, and the strongest published checkers are single models. We describe GauntletScore, an evidence engine whose few rules and thresholds were set on the benchmark's development split. It puts typed questions to independent readers from different providers and records each answer as a measurement in an evidence ledger. It combines the measurements with a fixed Bayesian rule that counts a contradiction only when two readers agree. On the full LLM-AggreFact test split (29,320 claims across 11 datasets), following the leaderboard's approach, GauntletScore achieves 79.51 balanced accuracy, above every average published on the leaderboard as of 15 November 2026. That places it level with the best of them, nimble-v4-max (79.45 from its published dataset scores; the engine's 95% interval, 78.47 to 80.53, contains it). On the same claims it beats two of its three readers used alone (Readers A and B) by 1.66 and 2.17 points (95% intervals 0.80 to 2.53 and 1.29 to 3.06). Without AggreFact-CNN, described below, the leads are 0.90 and 1.32 and both intervals stay above zero. The engine also records the evidence behind every verdict, which a single model does not. Its largest margin is on full-length news articles checked against multi-sentence summaries (AggreFact-CNN), a format that resembles much regulated writing. There it scores 76.45, the best on the leaderboard, against 70.1 for the next best and 63.9 to 67.8 for the six systems with the highest averages. The test split was run once, on an engine frozen before the first test claim, under a stopping plan committed in advance. We release per-claim predictions and the scoring script.
| Comments: | 10 pages, 1 figure, 7 tables. Per-claim predictions and scoring script: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.22027 [cs.CL] |
| (or arXiv:2611.22027v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22027 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ernie Hobbs [view email]
[v1]
Tue, 17 Nov 2026 00:38:54 UTC (120 KB)
来源:arXiv cs.AI · arxiv.org