arXiv cs.AI· Xuan Zhang, Zhijian Zhou, Zicheng Hu, Yan Feng, Hongyan Hao, Yuan Qi, Chao Qu·· 4 小时前AI 评分34
MCTS 数错了对象:为 LLM 推理学习探索计数
MCTS Is Counting the Wrong Thing: Learning Exploration Counts for LLM Reasoning
AI 导读
研究指出标准 UCT 为文本分支独立分配访问计数,导致同一推理策略的不同表述重复获得探索奖励、计数碎片化并增加累积遗憾。为此提出 LPC-MCTS,仅在 UCT 的探索统计量上改动,用轻量 Coin-Flipping Network 头基于策略隐藏表示估计覆盖度,让相关分支共享探索证据。
正文 · 原文
Abstract:Monte Carlo tree search (MCTS) has powered landmark successes in reinforcement learning, yet its gains in LLM reasoning can remain modest despite substantial search effort. We identify an overlooked bottleneck: what MCTS counts. Standard UCT assigns independent visit counts to textual branches, allowing different formulations of the same reasoning strategy to repeatedly receive exploration bonuses. Our theoretical analysis shows that this count fragmentation can impair exploration efficiency, as quantified by increased cumulative regret. To address this problem, we introduce Learned Pseudo-Count MCTS (LPC-MCTS), which changes only the exploration statistic in UCT. A lightweight Coin-Flipping Network head on the policy's hidden representations estimates representation coverage, enabling related branches to share exploration evidence. Across 5 mathematical-reasoning and 7 search benchmarks, LPC-MCTS consistently outperforms strong MCTS baselines. Holding each LPC-MCTS-trained policy fixed, replacing standard MCTS counts at inference improves average mean@16 accuracy across 5 mathematical benchmarks by 3.79 percentage points on Qwen2.5-7B and 2.71 points on Qwen3-4B. Applying LPC-MCTS during both training and inference yields gains of 4.96 and 4.37 points over standard MCTS, respectively. Together, our theory and experiments show that changing what MCTS counts can substantially improve its exploration efficiency and reasoning performance.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.22301 [cs.AI] |
| (or arXiv:2611.22301v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22301 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xuan Zhang [view email]
[v1]
Tue, 17 Nov 2026 06:57:40 UTC (1,619 KB)
来源:arXiv cs.AI · arxiv.org