arXiv cs.CL· Davi Bastos Costa, Renato Vicente·· 12 小时前AI 评分49
有害数据微调引发涌现性失配：研究提出“人格模型坍缩”机制
Persona-Model Collapse in Emergent Misalignment
AI 导读
研究提出“人格模型坍缩”是涌现性失配的一个方面：在有害数据上微调大语言模型，会削弱其内部编码和区分不同人格的表征。行为实验中，四个模型家族的有害微调版本在角色扮演问卷中给出更两极化的答案，同一人格下重复作答的一致性下降。机制分析显示，Qwen3-235B 失配版本中不同人格提示词激活的表征在末层占据的有效维度更少。
正文 · 原文
Abstract:Fine-tuning large language models on narrow data with harmful content produces broadly misaligned behavior on unrelated prompts, a phenomenon known as emergent misalignment. We propose that emergent misalignment involves persona-model collapse: a deterioration of the internal representations that encode and distinguish personas, undermining the nuance and consistency of their expression. We first test this hypothesis behaviorally, by comparing how models fine-tuned on harmful data and benign controls answer questionnaires while role-playing diverse personas. Across four model families, harmful fine-tuning produces more polarized, all-or-nothing answers, suggesting more binary, less nuanced persona expression. Histograms show increased concentration at the scale endpoints, and Kullback--Leibler (KL) divergence quantifies the distributional shift. We also find reduced within-persona consistency: repeated answers to the same question under the same persona become less stable. We reproduce these observations across diverse harmful datasets and questionnaires that probe different aspects of persona-conditioned behavior, while capability and simple instruction-following controls argue against a uniform loss of general ability. Finally, a mechanistic analysis of persona-conditioned activations in Qwen3-235B shows that representations elicited by different persona prompts span fewer effective dimensions in the misaligned variant than in the original and benign-control variants, most clearly in the final layers. Together, these findings provide behavioral and representational support for persona-model collapse as an aspect of emergent misalignment.
| Comments: | Accepted at the 40th Conference on Neural Information Processing Systems (NeurIPS 2026) |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.12850 [cs.CL] |
| (or arXiv:2605.12850v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.12850 arXiv-issued DOI via DataCite |
Submission history
From: Davi Bastos Costa [view email]
[v1]
Wed, 13 May 2026 00:48:57 UTC (206 KB)
[v2]
Sun, 24 May 2026 17:19:56 UTC (197 KB)
[v3]
Thu, 5 Nov 2026 17:23:22 UTC (811 KB)
来源：arXiv cs.CL · arxiv.org