arXiv cs.CL· Baris Askin, Muhammed Ustaomeroglu, Anupam Nayak, Gauri Joshi, Guannan Qu, Carlee Joe-Wong·· 10 小时前AI 评分41
研究数据与训练方法对涌现性失准与潜意识学习的影响
Examining the Impact of Data and Training Methods on Emergent and Subliminal Misalignment
AI 导读
微调语言模型会引发超出训练目标的行为变化，研究聚焦涌现性失准（EM）与潜意识学习（SL）两种现象。实验发现，EM 更容易迁移到与微调数据任务相同的提示词上，而非领域相同的提示词；SL 在 on-policy 蒸馏下同样发生失准迁移，强度高于 SFT、与 off-policy 蒸馏相当。教师模型固定时，数据决定学生继承失准的程度，教师则决定失准迁移到哪些领域和任务。
正文 · 原文
Abstract:Post-training a language model can induce surprising behavioral changes far beyond the intended training target. We study two particularly striking examples: emergent misalignment (EM), where fine-tuning on a narrow dataset leads to misaligned behavior in unrelated contexts, and subliminal learning (SL), where a student model inherits a teacher's misalignment through seemingly benign training data. We examine how the training data, evaluation data, and training method shape where and how strongly misalignment transfers between data and models. We characterize the data by the domain of a prompt (its topical context) and its task (the function the model performs), and we build natural-language and synthetic datasets organized as domain-task grids. We find that EM transfers more strongly to prompts that share the task of the fine-tuning data than to prompts that share its domain. In the synthetic setting, tasks that the model has learned well spread misalignment more broadly. For SL, we go beyond SFT (supervised fine-tuning) to off-policy distillation on the teacher's full next-token distribution and, to our knowledge for the first time, on-policy distillation, where the teacher only scores tokens sampled by the student. Misalignment transfers even under on-policy distillation, more strongly than under SFT and at a level comparable to off-policy distillation. Unlike SFT, where the teacher both generates the data and supervises the student, distillation lets us separate the two. With the teacher fixed, the data change how much misalignment the student inherits, but the teacher determines to which domains and tasks it transfers most strongly. Together, these results show how data and training influence the extent and patterns of emergent and subliminal misalignment.
| Comments: | Accepted to NeurIPS 2026 |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.12798 [cs.LG] |
| (or arXiv:2605.12798v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.12798 arXiv-issued DOI via DataCite |
Submission history
From: Baris Askin [view email]
[v1]
Tue, 12 May 2026 22:27:32 UTC (581 KB)
[v2]
Tue, 10 Nov 2026 05:41:12 UTC (750 KB)
来源：arXiv cs.CL · arxiv.org