arXiv cs.CL· Divyansh Singh·· 12 小时前AI 评分44
LeakScale：量化基准测试曝光对模型成绩的因果影响
LeakScale: Estimating the Causal Effect of Benchmark Exposure
AI 导读
LeakScale 是一个因果测量框架，通过构造依赖私有不可推导信息的新鲜可执行任务、反事实校验候选答案并训练互补适配器，量化基准曝光对成绩的影响。在 2,048 个任务族、两个模型族、两个可执行领域、三种任务形式和 589,824 次生成中，曝光使精确任务准确率提升 +3.22 至 +10.66 个百分点，所有族聚类 95% 区间均排除零，四项主检验通过 Holm 校正。
正文 · 原文
Abstract:Benchmark contamination is usually treated as a provenance question: did evaluation material appear in training? Provenance cannot supply the quantity needed to interpret a benchmark score: how much exposure changed it. We introduce LeakScale, a causal measurement framework that makes benchmark-specific exposure manipulable and measures its effect. LeakScale constructs fresh executable tasks whose correct behavior depends on private, non-derivable information, counterfactually certifies every candidate answer, and trains complementary adapters so that every family is exposed in exactly one adapter and unexposed in the other. Across 2,048 families, two model families, two executable domains, three task formulations, and 589,824 generations, exposure increases exact-task accuracy in every model-by-domain cell. Primary effects range from +3.22 to +10.66 percentage points; every family-clustered 95% interval excludes zero, and all four primary tests survive Holm correction. The effect also remains positive in all eight surface-paraphrase and isomorphic-transfer measurements. These results show that benchmark exposure can causally and reproducibly alter reported performance, including after the evaluation surface changes. LeakScale shifts contamination analysis from detecting contact to measuring how strongly a score depends on it.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.27176 [cs.CL] |
| (or arXiv:2609.27176v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.27176 arXiv-issued DOI via DataCite |
Submission history
From: Divyansh Singh [view email]
[v1]
Wed, 23 Sep 2026 00:28:57 UTC (7,779 KB)
[v2]
Thu, 24 Sep 2026 20:00:34 UTC (7,780 KB)
[v3]
Fri, 23 Oct 2026 05:37:08 UTC (55 KB)
来源：arXiv cs.CL · arxiv.org