arXiv cs.AI· Lena Libon, Alexander Panfilov, Ben Rank, Xin Chen, Jonas Geiping, Maksym Andriushchenko·· 13 小时前AI 评分55
研究：用探针引导微调对齐模型，同时保留可监控性
Alignment via Training Against Probes Without Losing Monitorability
AI 导读
论文研究探针引导微调，用检测模型激活中不良属性的探针作为直接训练信号，在无害性和诚实性两个对齐目标上评估线性与非线性探针。结果显示，训练中不更新的探针目标容易被利用，而持续更新的探针能显著降低有害性、提升诚实性并保持效用，在安全与效用的取舍上优于 DPO 和推理时引导，对越狱和 abliteration 攻击也更鲁棒。微调后相关概念仍以线性方式编码，说明该方法没有丢失可监控性。
正文 · 原文
Abstract:Models are usually aligned based on their observed outputs, using demonstrations, preference data, or reward signals. These objectives reward responses that look aligned. More capable models may learn to satisfy them without internalizing the intended behavior, for example by faking compliance during training. Such superficial compliance could be harder when the objective is defined on model internals rather than outputs. Therefore, we study probe-guided fine-tuning, using probes that detect undesired properties in model activations as a direct training signal. We evaluate linear and non-linear probes with different numbers of probes per layer across two alignment objectives: harmlessness and honesty. We find that training against probes that do not update during training is an easily exploitable objective, while continuously updated probes substantially reduce harmfulness and improve honesty while preserving utility. Probe-guided fine-tuning achieves better safety-utility trade-offs than DPO and inference-time steering, while being substantially more robust against jailbreak and abliteration attacks. Moreover, the concepts stay linearly encoded after fine-tuning, meaning oversight is not lost by our method. Training against probes thus offers a way to shape what models represent rather than only what they output, which may become increasingly important as models get better at making their outputs look aligned.
| Comments: | 39 pages, 22 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2609.38645 [cs.LG] |
| (or arXiv:2609.38645v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.38645 arXiv-issued DOI via DataCite |
Submission history
From: Lena Libon [view email]
[v1]
Tue, 29 Sep 2026 23:03:54 UTC (831 KB)
[v2]
Thu, 1 Oct 2026 13:38:27 UTC (831 KB)
[v3]
Tue, 20 Oct 2026 17:21:47 UTC (877 KB)
来源：arXiv cs.AI · arxiv.org