Amazon 论文审计 9 个政策决策基准，发现评分器和参考答案自身有错
Amazon 新论文发现，用于评测模型处理税务、法律和商业规则的基准常把正确答案判错，问题出在基准自身的参考答案或评分代码。研究团队审计了 9 个公开基准，通过回溯规则原文、重跑参考程序、用受控答案测试评分器，并在修复后对同一批已保存的模型回答重新打分。
Rohan Paul· @rohanpaul_ai · X·2026-11-09 05:24· 1 小时前AI 评分59
AI 导读
Amazon 论文审计 9 个政策类决策基准，发现用于评测模型处理税务、法律和商业规则的基准常把正确答案判错，原因在基准自身的参考答案或评分代码有误。
正文 · 原文
New Amazon paper finds that benchmarks used to grade AI models on tax, legal and business rules often mark correct answers wrong, because the tests’ own answer keys or grading code contain mistakes.
Benchmarks like RuleArena and LegalBench test whether models apply written rules to tax, legal, and business cases. If the reference answer or the scorer is wrong, the score measures the benchmark's bugs, not the model.
They audited 9 public benchmarks by tracing references back to the source rules, rerunning reference programs, and testing scorers with controlled answers. Then they rescored the same saved model responses after each fix.
LegalBench's tax scorer deleted decimal points, so $259,487.13 was read as 25,948,713. Fixing it raised accepted answers from 8 or 9 to between 87 and 91 out of 96 for each of 3 models.
Before trusting a low score on a rule-following benchmark, read the scorer code and check a few failed answers by hand.
来源：Rohan Paul · x.com
来源：RadarAI · aihot.news