跳到正文
arXiv cs.CL· Abhishek Anand·· 2 天前AI 评分20

部分提示词污染测试的假阳性与假阴性:剩余文本长度如何影响基准污染检测

How Much Is Left Matters: False Positives and False Negatives in Partial-Prompt Contamination Tests

AI 导读

一项针对部分提示词补全式基准污染检测的审计发现,记忆标记强烈依赖剩余文本长度:在 Qwen2.5-7B-Instruct 上,剩余不超过 8 个词的问题被标记比例为 31.5%,而剩余至少 25 个词时仅 5.5%。

正文 · 原文

This paper has been withdrawn by Abhishek Anand

No PDF available, click to view other formats

Abstract:Partial-prompt completion, where a model receives the first part of a benchmark problem and is checked for whether it reproduces the rest, is a widely used test for benchmark contamination. We audit this test on seven open-weight models using MATH-500 and recent AIME 2025/2026 problems, which serve as true negatives. Memorization flags depend strongly on how much text remains: for every model, problems with at most 8 leftover words are flagged several times more often than problems with at least 25 (31.5% vs. 5.5% for Qwen2.5-7B-Instruct), and half to three-quarters of all ROUGE-based flags come from short leftovers. On true negatives, a ROUGE-L >= 0.75 threshold flags up to 6.7% of problems at a 60% prefix, and at an 80% prefix even exact match flags 8.6% (pooled over models; 6.2% on AIME 2026, which postdates every model), against 0% at 40-60%. The test also misses contamination upstream: models distilled from Qwen2.5-Math, whose base is heavily flagged in prior work, show almost no model-specific recall. Calibrating each model against models from other families removes most flags while isolating a consistent Qwen2.5-specific signal at every prefix length. We recommend length-stratified reporting, prefixes of at most 60%, style-matched post-cutoff controls, and cross-family calibration.
Comments: The false-negative analysis in Section 4 used a model whose pretraining data cannot serve as ground truth for exposure, and the mechanism discussed in Section 4.3 is incomplete. Please do not cite this version
Subjects: Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as: arXiv:2611.10307 [cs.LG]
  (or arXiv:2611.10307v3 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2611.10307

arXiv-issued DOI via DataCite

Submission history

From: Abhishek Anand [view email]
[v1] Fri, 6 Nov 2026 23:13:57 UTC (33 KB)
[v2] Wed, 11 Nov 2026 21:10:54 UTC (1 KB) (withdrawn)
[v3] Fri, 13 Nov 2026 05:54:07 UTC (1 KB) (withdrawn)

来源:arXiv cs.CL · arxiv.org