arXiv cs.CL· Johanna Angulo, V\'ictor Yeste, Hector Espinos-Morato·· 11 小时前AI 评分32
按可击败的缓解措施分类的基准污染分类法
Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation
AI 导读
研究者提出按“可击败的缓解措施”组织的基准污染分类法，分为直接、派生、时序、分布与评测中习得五类，覆盖训练期与评测期泄漏，私有测试集只能堵住第一类。第五类在评测过程中产生，须随报告分数一并记录，并被落实为含“unknown”合法取值的四字段披露协议，以 CC BY 4.0 发布并附 JSON Schema、校验器与示例。
正文 · 原文
Abstract:A benchmark score is a joint property of the model, the evaluation harness, the elicitation budget, the sampled population, and contamination status. Leaderboards publish the model and the score, so capability and leakage stay observationally equivalent. Existing taxonomies classify contamination for automated detection, not the question a reporter faces at publication: given the mitigations already applied, which validity threats remain open? We introduce a taxonomy organized by the mitigation each type defeats -- direct, derivative, temporal, distributional, and acquired -- spanning training-time and evaluation-time leakage. Holding out a private test set closes the first alone. The fifth is acquired during the evaluation itself; because it is a property of one run, it must be recorded with the reported score rather than with the benchmark release. We operationalize it as a four-field disclosure protocol in which "unknown" is a valid entry, released under CC BY 4.0 with a JSON Schema, a validator, and worked examples. Two coders external to the design team applied a pre-registered instrument to 41 documents. Per-variable linear-weighted $\kappa$ runs from 0.00 to 0.35 (median 0.21) over 29 main-pass documents against a single-coder test-retest ceiling of 0.84, collapsing under the class skew the registration anticipated; pooling raises it to 0.46 through chance correction rather than better agreement. Two variables fall below the prevalence-robust threshold registered in advance: strata reporting and the acquired type introduced here. Disagreement concentrates on when a variable applies rather than on what a document states. Elicitation budgets are reported in 13% of documents, and no document addresses all five types. The contribution is the taxonomy, the score-side artifact that follows from it, and a pre-registered measurement of instrument reliability and current disclosure.
| Comments: | 18 pages, 3 figures, 6 tables. Specification, audit instrument, pre-registration and analysis code: this https URL |
| Subjects: | Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2608.29463 [cs.CR] |
| (or arXiv:2608.29463v2 [cs.CR] for this version) | |
| https://doi.org/10.48550/arXiv.2608.29463 arXiv-issued DOI via DataCite |
Submission history
From: Johanna Angulo [view email]
[v1]
Sat, 29 Aug 2026 23:04:09 UTC (37 KB)
[v2]
Fri, 6 Nov 2026 13:39:00 UTC (46 KB)
来源：arXiv cs.CL · arxiv.org