arXiv cs.AI· Zechuan Wang, Siyuan Lu, Hongxuan Zhang, Linjian Mo, Chenyi Zhuang, Leilei Gan·· 12 小时前AI 评分29
CrEST：面向多轮多步 LLM 智能体的验证器锚定信用分配框架
Teach the Magnitude, Not the Direction: Verifier-Anchored Credit Assignment for Multi-Turn Multi-Step LLM Agents
AI 导读
CrEST 提出一种分层信用分配框架，将每个采样 token 的标量优势符号锚定到验证器奖励，同时用特权自教师提供稠密幅度信号。它在轮次级用验证优势缓解轮间稀释，在轮内用熵门控自教师调制细化 token 贡献。在 BFCL V3 和 WildToolBench 上，CrEST 在两个模型规模下均优于 RL 与蒸馏基线，长轨迹和严格会话级指标提升最大。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards (RLVR) directly optimizes verifier-defined outcomes for multi-turn tool-use agents, yet its trajectory-level credit assignment conflates heterogeneous per-turn outcomes into a single reward signal. On-policy distillation provides dense per-token supervision but uses a teacher-defined token-level objective and can be prone to gradient concentration collapse. We introduce $\textbf{CrEST}$, a hierarchical credit assignment framework that anchors each sampled token's scalar advantage sign to verified rewards while incorporating dense magnitude signals from a privileged self-teacher. $\textbf{CrEST}$ resolves credit at two levels: turn-segmented verified advantages address inter-turn dilution, while entropy-gated self-teacher modulation refines intra-turn token contributions. Experiments on BFCL V3 and WildToolBench show that $\textbf{CrEST}$ consistently outperforms both RL and distillation baselines across two model scales, with the largest gains on long-trajectory and strict session-level metrics. Our work demonstrates that the teacher's role in policy optimization can be reduced from determining token-level reinforcement or suppression to modulating update magnitudes, enabling dense credit assignment while keeping verifier rewards as the training objective.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.13179 [cs.AI] |
| (or arXiv:2608.13179v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2608.13179 arXiv-issued DOI via DataCite |
Submission history
From: Siyuan Lu [view email]
[v1]
Thu, 13 Aug 2026 12:44:37 UTC (12,074 KB)
[v2]
Fri, 16 Oct 2026 14:02:05 UTC (12,077 KB)
来源：arXiv cs.AI · arxiv.org