arXiv cs.AI· Yunan Zeng, Jinhao Jing, Xi Cheng, Yuwang Wang·· 12 小时前AI 评分24
ASE：用激活空间引导为可验证奖励强化学习增加探索
Exploration via Activation-Space Steering for Reinforcement Learning with Verifiable Rewards
AI 导读
研究者提出 Activation-Space Exploration（ASE），一种在线多样化训练 rollout 的方法：通过聚类同一问题正确与错误回答之间的激活差异，构建潜在探索向量库，并在后续 rollout 中施加不同向量以探索其他推理路径。
正文 · 原文
Abstract:Reinforcement learning with verifiable rewards (RLVR) trains a language model by sampling responses, scoring them with an automatic verifier, and updating the policy from the resulting rewards. The policy therefore receives direct learning signals only from trajectories that appear in its rollouts. Yet repeated sampling from the same policy can concentrate on similar reasoning paths, leaving useful alternatives unexplored under a fixed rollout budget. We introduce Activation-Space Exploration (ASE), an online method for diversifying training rollouts. By clustering activation differences between correct and incorrect responses to the same problem, ASE builds a structured bank of latent exploration vectors. Different vectors are then applied to subsequent rollouts to explore alternative reasoning paths. On Qwen3-1.7B-Base and Qwen3-8B-Base trained on DAPO-Math-17k, ASE improves average Mean@8 across six mathematical reasoning benchmarks over GRPO by 4.87 and 1.87 percentage points, respectively, under the same rollout budget.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.27012 [cs.LG] |
| (or arXiv:2610.27012v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.27012 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yunan Zeng [view email]
[v1]
Mon, 19 Oct 2026 12:23:01 UTC (841 KB)
来源：arXiv cs.AI · arxiv.org