arXiv cs.CL· Danlin Li, Miao Li, Xu Han, Jiecheng Lu·· 3 天前AI 评分22
目标 token 不对称如何伪装成语言模型的深度特化
Target-Token Asymmetry Can Masquerade As Depth Specialization
AI 导读
层消融研究常把跨语料库的早层损伤差异解读为任务特化，但 arXiv 论文指出这可能只是目标 token 类型不同造成的假象。在 GSM8K 与 Wikitext 之间，正交互效应在五个模型实例上复现，而三个完全分解的模型中只有空白字符呈正交互；移除空白字符后三者汇总效应全部反转，匹配完整目标混合后两个模型反转。
正文 · 原文
Abstract:Layer-ablation studies often perturb early or late layers of a language model, compare the damage to prediction on two corpora, and read a larger early-layer effect on one corpus as early-layer specialization for its task. Such a comparison averages over the tokens that each corpus asks the model to predict, and these target tokens differ: math solutions, for example, ask for far more whitespace and numbers than encyclopedia prose. We show that this difference can create the appearance of depth specialization. We split a cross-corpus early--late damage interaction by target-token type and recompute it under a target mixture shared by both corpora. Between GSM8K and Wikitext, a positive interaction replicates across five model instances. In the three fully decomposed models, only whitespace has a positive interaction. Removing whitespace reverses the pooled effect in all three, and matching the full target mixture reverses it in two. The diagnosis holds under position matching, adversarial reassignment of token types, and a continuous readout. The risk can also be anticipated from the tokenizer alone. A screen computed before any model runs places every held-out pair it scored in the region its label predicts, and on the pairs it rates at high or medium risk, removing whitespace reverses every positive pooled estimate. A smaller content-word effect one token ahead follows the share of content targets that the unablated model already predicts. A cross-corpus depth effect therefore supports an interpretation of task specialization only after the target-token mixture is reported, the effect is decomposed by token type, and a mixture-matched summary keeps its sign.
| Comments: | 48 pages, 7 figures |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19344 [cs.CL] |
| (or arXiv:2610.19344v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19344 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Danlin Li [view email]
[v1]
Wed, 14 Oct 2026 00:36:45 UTC (802 KB)
来源：arXiv cs.CL · arxiv.org