arXiv cs.AI· Julia Barth, Armin Sommer, Kunvar Thaman, Jannik Schilling·· 14 小时前AI 评分37
目标去哪了？目标泛化失败中策略对齐的三种不同失效深度
What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization
AI 导读
研究在八项视觉导航与操作任务中发现，目标误泛化时目标信息会在三个不同深度上丢失，且丢失深度决定哪种修复方式有效。目标被读取但被代理目标压制时，引导现有策略可将目标到达率从 16.0% 提升至 95.3%；目标被读取但未被遵循时，轻量下游接线可将到达率从 31.6% 恢复至 84.3%；目标完全未被读取时所有方法均失效。作者据此主张目标误泛化应在智能体内部而非仅从行为层面评估。
正文 · 原文
Abstract:Goal misgeneralization is a common failure mode in which an agent behaves competently out of distribution but pursues a proxy that coincided with the intended goal during training instead of the goal itself. However, we still have a weak understanding of what happens to the intended goal in misgeneralizing agents and what enables them to be recovered without retraining. We study this across eight visual navigation and manipulation tasks, including robot control. While tracing where goal information remains available internally, how strongly it influences the policy, and what role it plays in action selection, we uncover that the goal is lost at one of three different depths. This depth predicts which recovery works in each task. When the goal is followed, but outcompeted by the proxy, steering the existing policy redirects it to the goal's location, raising goal reach from 16.0% to 95.3%. When it is read, but not followed, steering fails, but a lightweight new downstream wiring recovers goal reach from 31.6% to 84.3%. When it is not read, none of the methods work, as the policy does not use the goal information. The same tracing also reveals what replaces the goal: functional target substitution, where behavior follows an alternative goal, a policy shortcut, where behavior follows a learned action strategy, or cue competition between intended and proxy goal cues. Knowing where the goal is lost tells us whether an agent can be repaired or has to be retrained. Goal misgeneralization should therefore be studied and evaluated inside the agent, not only from its behavior.
| Comments: | 10 pages main text, 122 pages total. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO) |
| Cite as: | arXiv:2610.19731 [cs.LG] |
| (or arXiv:2610.19731v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19731 arXiv-issued DOI via DataCite |
Submission history
From: Julia Barth [view email]
[v1]
Wed, 14 Oct 2026 07:52:24 UTC (8,556 KB)
[v2]
Wed, 28 Oct 2026 01:02:24 UTC (8,558 KB)
来源：arXiv cs.AI · arxiv.org