arXiv cs.CL· Yanru Xiao, Yifei Cai·· 11 小时前AI 评分38
审计发现九个政策决策基准存在评分错误，提出 BAR 修订方案
Follow the Policy, Fail the Benchmark: Auditing and Revising Policy-Based Decision Benchmarks
AI 导读
研究者审计九个公开政策决策基准，发现存在无依据的目标答案、重放后仍存活的参考错误，以及拒绝可接受答案或给不完整决策记分的评估器。在 LegalBench SARA 的 96 个数值案例上，仅修正评分器就让三个模型的接受率各提升超过 80 个百分点；RuleArena Tax 中恢复必需的零下限改动了 300 条参考中的 27 条。
正文 · 原文
Abstract:Policy-based decision benchmarks evaluate whether systems apply written rules to financial and operational cases. Yet a rule-following response can be marked wrong when the benchmark misstates the required decision. We audit nine public policy-based decision benchmarks through source tracing, program execution, and controlled checks. We find unsupported targets, reference errors that survive replay or propagate as solved steps, and evaluators that reject acceptable answers or credit incomplete decisions. To measure the scoring impact, we rescore unchanged model responses. On the 96 LegalBench SARA numeric cases, correcting the scorer raises acceptance by more than 80 percentage points for each of three models, without changing references or responses. In RuleArena Tax, restoring a required zero floor changes 27 of 300 references and narrows an observed model gap. A controlled federal-exemption task with explicit ownership and eligibility assumptions also shows that errors in supplied intermediate answers can affect final decisions. We present BAR (Benchmark Audit and Revision), a versioned collection of audit evidence and revisions for reevaluation.
| Comments: | 28 pages. Code and revised data: this https URL |
| Subjects: | Computation and Language (cs.CL); Software Engineering (cs.SE) |
| Cite as: | arXiv:2611.07042 [cs.CL] |
| (or arXiv:2611.07042v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07042 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yanru Xiao [view email]
[v1]
Wed, 4 Nov 2026 19:04:55 UTC (136 KB)
来源：arXiv cs.CL · arxiv.org