arXiv cs.AI· Juan-Pablo Rivera, Ari Spiesberger, Juan J Vazquez, Arun Jose, Axel Ahlqvist, Adeline Kassler, Meru Gopalan, Sandy Tanwisuth, John Hughes·· 11 小时前AI 评分40
共谋模型生物：不可信监控器中的纠正迁移
Model Organisms of Collusion: Corrective Transfer in Untrusted Monitors
AI 导读
研究对 Llama-3.3-70B 和 Qwen2.5-72B 植入五组策略-监控行为对，仅用 300 条已捕获共谋的诚实评审纠正监控器，可将未见提示格式下的其余四类共谋从 91% 降至 14-20%，而用等量无关样本微调仍残留 56-93%。已捕获行为本身几乎被完全消除（残留 1-2%），但未知共谋仍有 14-20% 存续，说明对已知失败的近乎完美纠正会高估共谋的实际清除程度。
正文 · 原文
Abstract:AI control protocols aim to get useful work from models that may be misaligned by monitoring their behaviour. One approach uses a second copy of the same model as the monitor. This avoids relying on a weaker reviewer the model could outsmart, but risks collusion: the model, now monitoring itself, may cover up its own misbehaviour. One remedy is to fine-tune the monitor on whatever examples of misbehaviour the control team has caught. We ask whether this training also removes collusion that the control team has not yet found. We fine-tune Llama-3.3-70B and Qwen2.5-72B to install five policy-monitor behaviour pairs, correct the monitor on 300 honest reviews of one caught collusion, and measure collusion on the other four in unseen prompt formats. Averaged over the three seeds of each family, correction reduces held-out collusion from 91% to 14-20% under three of the four methods we compare (individual seeds 10-32%), while a control fine-tuned on the same number of unrelated examples leaves 56-93%. Family-average accuracy on control items changes by at most five points and capability is preserved, apart from a drop in mathematical accuracy on one Llama seed. Under the tested training configuration, methods with similar caught-task accuracy differ in held-out transfer, from 58% to 83% average reduction, so the caught behaviour alone does not reveal how far a correction reaches. Training on a known failure removes it almost entirely (1-2% remaining), yet 14-20% of unknown failures persist, so near-perfect correction of the caught behaviour overstates how much collusion has been removed.
| Comments: | 33 pages, 6 figures, 15 tables |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.30582 [cs.AI] |
| (or arXiv:2610.30582v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30582 arXiv-issued DOI via DataCite |
Submission history
From: Juan-Pablo Rivera [view email]
[v1]
Thu, 22 Oct 2026 01:22:48 UTC (681 KB)
[v2]
Fri, 30 Oct 2026 17:55:18 UTC (681 KB)
来源：arXiv cs.AI · arxiv.org