arXiv cs.CL· Maolin Ran, Minda Chen, Di Huang, Wei Xiong, Fei Huang, Weiwen Liu·· 3 天前AI 评分24
DGS-WM：面向语言智能体的决策引导选择性世界建模
Model What Matters: Decision-Guided Selective World Modeling for Language Agents
AI 导读
针对语言智能体强化学习中奖励稀疏、缺乏对动作后果显式监督的问题，研究者提出 DGS-WM（决策引导选择性世界建模），通过决策引导观测选择筛选有决策价值的观测单元作为世界建模目标，并用置信度自适应加权调整监督强度，与 RL 损失在同一策略模型上联合优化。在单跳/多跳问答和交互式网页任务等多个 benchmark 上，DGS-WM 在不同模型规模和 RL 算法下均取得一致提升，优于全观测世界建模方法。
正文 · 原文
Abstract:Reinforcement learning provides a direct mechanism for optimizing task objectives in large language model-driven agents, but task rewards are typically sparse trajectory-level signals that only evaluate task success, without directly supervising action-conditioned environment transitions, leaving the model with limited explicit understanding of action consequences and environment dynamics. Recent work introduces auxiliary supervision through policy and world modeling co-training, yet typically treats complete environmental observations uniformly as world modeling targets, overlooking the varying decision relevance of observation content. We propose DGS-WM (Decision-Guided Selective World Modeling), a decision-guided policy-world modeling co-training approach for language agents. DGS-WM employs a Decision-Guided Observation Selection component that filters observation units with decision value as world modeling targets based on their support for subsequent policy responses, and a Confidence-Adaptive Weighting component that adaptively adjusts supervision strength according to the model's prediction confidence on selected targets. Together, these components form a decision-aware auxiliary world modeling loss, jointly optimized with the RL loss on the same policy model. Across multiple benchmarks covering single-hop/multi-hop question answering and interactive web tasks, DGS-WM achieves consistent improvements under different model scales and RL algorithms, demonstrating more stable and effective performance compared to full-observation world modeling methods.
| Comments: | Core implementation included as ancillary files |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19642 [cs.AI] |
| (or arXiv:2610.19642v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19642 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Maolin Ran [view email]
[v1]
Wed, 14 Oct 2026 06:11:08 UTC (320 KB)
来源：arXiv cs.CL · arxiv.org