arXiv cs.CL· Abhishek Anand·· 2 天前AI 评分20
部分提示词污染测试的假阳性与假阴性:剩余文本长度如何影响基准污染检测
How Much Is Left Matters: False Positives and False Negatives in Partial-Prompt Contamination Tests
AI 导读
一项针对部分提示词补全式基准污染检测的审计发现,记忆标记强烈依赖剩余文本长度:在 Qwen2.5-7B-Instruct 上,剩余不超过 8 个词的问题被标记比例为 31.5%,而剩余至少 25 个词时仅 5.5%。
正文 · 原文
This paper has been withdrawn by Abhishek Anand
No PDF available, click to view other formats
Abstract:Partial-prompt completion, where a model receives the first part of a benchmark problem and is checked for whether it reproduces the rest, is a widely used test for benchmark contamination. We audit this test on seven open-weight models using MATH-500 and recent AIME 2025/2026 problems, which serve as true negatives. Memorization flags depend strongly on how much text remains: for every model, problems with at most 8 leftover words are flagged several times more often than problems with at least 25 (31.5% vs. 5.5% for Qwen2.5-7B-Instruct), and half to three-quarters of all ROUGE-based flags come from short leftovers. On true negatives, a ROUGE-L >= 0.75 threshold flags up to 6.7% of problems at a 60% prefix, and at an 80% prefix even exact match flags 8.6% (pooled over models; 6.2% on AIME 2026, which postdates every model), against 0% at 40-60%. The test also misses contamination upstream: models distilled from Qwen2.5-Math, whose base is heavily flagged in prior work, show almost no model-specific recall. Calibrating each model against models from other families removes most flags while isolating a consistent Qwen2.5-specific signal at every prefix length. We recommend length-stratified reporting, prefixes of at most 60%, style-matched post-cutoff controls, and cross-family calibration.
| Comments: | The false-negative analysis in Section 4 used a model whose pretraining data cannot serve as ground truth for exposure, and the mechanism discussed in Section 4.3 is incomplete. Please do not cite this version |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.10307 [cs.LG] |
| (or arXiv:2611.10307v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.10307 arXiv-issued DOI via DataCite |
Submission history
From: Abhishek Anand [view email]
[v1]
Fri, 6 Nov 2026 23:13:57 UTC (33 KB)
[v2]
Wed, 11 Nov 2026 21:10:54 UTC (1 KB) (withdrawn)
[v3]
Fri, 13 Nov 2026 05:54:07 UTC (1 KB) (withdrawn)
来源:arXiv cs.CL · arxiv.org