arXiv cs.CL· Minqian Liu, Yuexi Shen, Heajun An, Qi Zhang, Pamela J. Wisniewski, Sang Won Lee, Jin-Hee Cho, Lifu Huang·· 14 小时前AI 评分29
Probe-RPO：用隐含信念作为可扩展过程奖励训练 LLM 策略性说服
Implicit Beliefs as Scalable Process Rewards for Strategic Persuasion
AI 导读
研究者提出 Probe-RPO，一种在线强化学习框架，通过从冻结的被说服者模型 logits 中读取三个内省探针，将对话对方内部状态转化为稠密过程奖励，用于 LLM 多轮策略性说服训练。
正文 · 原文
Abstract:Training large language models (LLMs) for multi-turn strategic persuasion faces two coupled obstacles: (i) realistic simulation of psychologically grounded persuadees, and (ii) dense, faithful supervision for credit assignment in strategic dialogues. Outcome-only rewards are sparse and uninformative about which conversational moves drove success, while LLM-judge process rewards are expensive and decoupled from the underlying interaction dynamics. We address both challenges with a unified framework. First, we introduce PersuArena, a persuasion simulation environment that produces realistic, non-monotonic trajectories. Second, building directly on this environment, we propose Process Rewards via Online Belief Elicitation for Relative Policy Optimization (Probe-RPO), an online RL framework that converts the persuadee's internal state into dense process rewards. At each turn, three introspective probes are read from the frozen persuadee's logits, yielding a multi-dimensional reward signal that is faithfully aligned with persuasion dynamics. With extensive experiments on PersuArena spanning healthcare, financial advising, workplace management, education, and personal relationships, we demonstrate that our Probe-RPO achieves the full acceptance rate of 52.5%, compared with the 35.8% for the strongest GRPO ablation and 25.8% for the best off-the-shelf LLM. We also conduct comprehensive analyses on the generalization across persuadee models and the persuasion dynamics across conversations. Our work offers a promising paradigm that can provide effective process supervision for strategic dialogue.
| Comments: | NeurIPS 2026 Workshop (Large Language Models for Social Reasoning and Simulation) |
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.33527 [cs.LG] |
| (or arXiv:2610.33527v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33527 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Minqian Liu [view email]
[v1]
Sat, 24 Oct 2026 08:08:23 UTC (266 KB)
来源：arXiv cs.CL · arxiv.org