arXiv cs.CL· Unnati Shah, Samruddhi Kakade, Aum Patel·· 11 小时前AI 评分39
LLM 评审到底核实了什么？文档时效性核查中的匹配对照与记录缺失
What Do LLM Judges Actually Verify? Matched Controls and Record Gaps for Document Currency
AI 导读
研究用 IETF RFC 索引中 48 处替换构造匹配答案，测试 LLM 评审能否依据修订记录判断引用文档是否仍有效。
正文 · 原文
Abstract:LLM judges often check, against an amendment record, whether the documents an answer cites are still in force. Such records are rarely complete. From a table of replacements, our test generates matched answers citing a replaced document, its successor, the replaced document before its replacement, and an older document that a newer one updated without replacing; it removes links from the record and scores each finding by its stated reason. On 48 replacements from the IETF RFC index, judges given a correct record caught nearly every replaced citation, as a lookup would. On the 24 faults whose link we removed, every judge caught more from the texts alone than with the record: Gemini 3.1 Pro 23/24 against 1/24, GPT-5.5 23/24 against 9/24, Claude Opus 4.5 17/24 against 9/24 and Gemma 4 26B 13/24 against 0/24. The drop recurred on a second draw of removed links. Most misses carried no finding; in an exploratory run with a conflict code, only Gemini used it often. Rewording the gap as "no replacement recorded" did not help; presenting the record as an unverified note, or warning that it was incomplete, restored most lost catches for Gemini, GPT-5.5 and Claude (significant after correction for the first two) but at most 5 of 13 for Gemma; where catches returned, so did false alarms on updates. With both texts, judges mostly told replacements from updates; every update flagged had a near-identical title. A descriptive case study on Indian rooftop-solar rules is appended; we release the harness.
| Comments: | 37 pages, 6 figures, including a 13-page supplementary case study. Code, data and all judge replies: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.01311 [cs.AI] |
| (or arXiv:2611.01311v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.01311 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Unnati Bhaven Shah [view email]
[v1]
Sat, 31 Oct 2026 22:19:54 UTC (249 KB)
来源：arXiv cs.CL · arxiv.org