arXiv cs.AI· Helena Merker, Nick Walker, Andreea Bobu·· 2 天前AI 评分24
让机器人主动提问：面向欠指定奖励的定向主动偏好学习
Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards
AI 导读
针对人类示教只能覆盖部分任务特征、导致奖励欠指定的问题，研究者提出把纠错负担从人转移到机器人：利用示教数据中暴露的监督薄弱特征构建先验，将偏好查询定向到这些特征，并为每次查询附上解释以聚焦人类注意力。仿真中该先验在低查询量场景下比标准主动偏好学习基线最高提升 30.8%，用户研究显示先验与解释各自显著改善奖励恢复且相互放大。该工作已被 CoRL 2026 接收。
正文 · 原文
Abstract:Reward learning typically assumes the human's input covers everything that matters for the task. In practice, humans can only attend to so much at once, so their teaching naturally captures some features---or aspects of the task---better than others. Prior work has shown how the robot can identify which features were poorly supervised directly from its teaching data. But the current solution is to ask the human to try again, putting the burden on someone who already struggled to convey those features. We propose shifting this burden to the robot. Our key insight is that the same information that tells the robot what it doesn't know can help it ask better questions. Preference queries offer a low-effort channel for resolving this uncertainty, but only if the robot knows where to direct them and the human knows what to attend to when answering. We act on this insight in two ways: we build an informed prior that targets preference queries at the features that need resolving, and we accompany each query with an explanation that focuses the human's attention on exactly those features. In simulation, the informed prior beats the standard active preference learning baseline by up to $30.8\%$ in the low-query regime, where reducing human effort matters most. In a user study, the informed prior and explanations each significantly improve reward recovery and amplify each other.
| Comments: | Accepted to the 10th Conference on Robot Learning (CoRL 2026), Austin, TX. 18 pages, 8 figures, 1 table |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.22285 [cs.RO] |
| (or arXiv:2610.22285v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22285 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nick Walker [view email]
[v1]
Thu, 15 Oct 2026 17:41:46 UTC (4,211 KB)
来源：arXiv cs.AI · arxiv.org