基于理由引导的人类奖励建模：面向偏好强化学习的 PbRL 框架

Reason-guided Human Reward Modeling for Preference-based Reinforcement Learning

AI 导读

一篇论文提出理由引导的偏好强化学习（PbRL）框架，从轨迹级反馈中学习人类奖励模型，除偏好标签外还引入解释行为为何被偏好的理由标签，使学到的奖励同时捕捉人类选择及其依据。相比仅用偏好的奖励学习，该形式提供额外监督以学习对齐人类的奖励，并提升策略可解释性。学习到的人类奖励在策略优化中与环境奖励结合，使智能体在提升任务表现的同时产生更可解释、更符合人类偏好且对偏好敏感的行为。

正文 · 原文

View PDF HTML (experimental)

Abstract:Reinforcement learning (RL) policies optimized for task rewards may produce behaviors that are efficient but misaligned with human expectations, safety preferences, or intent. This paper presents a reason-guided preference-based reinforcement learning (PbRL) framework that learns a human reward model from trajectory-level feedback. In addition to preference labels, the framework incorporates reason tags explaining why a behavior is preferred, enabling the learned reward to capture both human choices and their underlying rationale. Compared with preference-only reward learning, the reason-guided formulation provides additional supervision for learning human-aligned rewards and improves the interpretability of the resulting policy. The learned human reward is combined with the environment reward during policy optimization, allowing the agent to improve task performance while producing more interpretable, human- aligned, and preference-sensitive behavior

Subjects:	Robotics (cs.RO); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2610.23550 [cs.RO]
	(or arXiv:2610.23550v1 [cs.RO] for this version)
	https://doi.org/10.48550/arXiv.2610.23550 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Anshul Nayak [view email]
[v1] Fri, 16 Oct 2026 14:41:13 UTC (1,329 KB)

来源：arXiv cs.AI · arxiv.org