arXiv cs.AI· Austin Meek, Kyle Cox, Matthew Nguyen, Austin J. Brockmeier, Steven Basart·· 13 小时前AI 评分42
通过表征工程在真实场景中缓解奖励黑客行为
Mitigating Reward Hacking in Realistic Scenarios Through Representation Engineering
AI 导读
研究团队对三种方法获得的引导方向施加动态干预，以监测并缓解工具调用和长上下文等真实场景中的奖励黑客行为，包括基于标注 rollout 的 difference-in-means 向量、梯度优化向量，以及模型引用反奖励黑客模型规范时提取的向量。
正文 · 原文
Abstract:Reward hacking is a major problem in current production machine learning systems, resulting in costly and dangerous failures. Here we study how to monitor for and mitigate reward hacking in realistic settings, including tool calls and long contexts. We apply a dynamic intervention to steering directions obtained by three different methods: difference-in-means vectors from labeled rollouts, gradient optimization-based vectors, and vectors extracted when models reference an anti-reward-hacking model spec. This dynamic intervention perturbs model representations less than uniform steering vector addition to the residual stream, allowing for more precise edits. We find that we are able to cheaply and significantly reduce reward hacking without correspondingly significant costs to model coherence or capability, and that steering vectors originally extracted from coding environments also generalize to reducing reward hacking in other domains. Our results pave the way towards latent space monitoring and control over realistic production systems.
| Comments: | Project page: this https URL Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.29129 [cs.LG] |
| (or arXiv:2610.29129v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29129 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Austin Meek [view email]
[v1]
Wed, 21 Oct 2026 03:12:22 UTC (827 KB)
来源：arXiv cs.AI · arxiv.org