arXiv cs.CL· Yun Li, Ehsan Javanmardi, Simon Thompson, Jiachen Gong, Qunli Zhang, Zifan Zeng, Shiming Liu, Peng Wang, Zixuan Guo, Manabu Tsukada·· 13 小时前AI 评分38
ECHO：偏好优化中的共模隐藏偏移如何导致词表坍缩
ECHO: Emergent Common-Mode Hidden Offsets Drive Vocabulary Collapse in Preference Optimization
AI 导读
偏好优化可在所有排序指标提升的同时积累一种跨 token 位置、提示词及 chosen/rejected 响应的共模隐藏偏移，该偏移投影到输出层形成“词表井”，抬高单个 token 概率直至生成坍缩。
正文 · 原文
Abstract:Preference optimization can improve every ranking metric while building a hidden, shared representation shift that the relative loss barely sees, one that can eventually destroy generation. We trace this failure to the common-mode offset, a shared directional shift in the model's hidden states that appears across token positions, prompts, and both chosen and rejected responses. Because it moves chosen and rejected representations together, the relative ranking loss barely reacts to its accumulation. As this offset grows, it projects onto the output layer to form a vocabulary well, elevating a single token's probability until it dominates decoding and causes generation collapse. The isolated offset alone reconstructs the vocabulary well, injecting the learned offset into an untreated base model recreates the collapse where norm-matched random directions do not, and masking this common component from the loss prevents the well in our flagship setting. Under the per-token ranking loss we study, the same collapse appears in 14 of 17 base-model lineages and under full-parameter fine-tuning. Moreover, the offset is not merely an artifact of collapse, since it is already present in textbook DPO and SimPO runs that form no well. Textbook DPO keeps it weakly aligned with any single vocabulary direction even with three times our base budget. Trained from base models for that budget, however, full-parameter SimPO drives it into collapse on both Llama-2 and Llama-3.1, and the same kind of offset grows, without a well, from a public SFT checkpoint under the official SimPO recipe. Across 17 UltraFeedback runs, the better a run ranks held-out responses, the less often two public reward models prefer its samples to the base model's. These findings expose a blind spot in standard ranking metrics and introduce a representation-level signal that can warn of generation collapse before it occurs.
| Comments: | 28 pages, 13 figures |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.29410 [cs.CL] |
| (or arXiv:2610.29410v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29410 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yun Li [view email]
[v1]
Wed, 21 Oct 2026 08:35:39 UTC (362 KB)
来源：arXiv cs.CL · arxiv.org