arXiv cs.AI· Jingjie Ning, Xueqi Li, Yibo Kong·· 12 小时前AI 评分34
SearchHorizonBench：评估 AI 智能体搜索价值的基准
SearchHorizonBench: A Benchmark for Search Value in AI Agents
AI 导读
研究者推出 SearchHorizonBench，用于从执行经验中学习和评估 AI 智能体的搜索价值，包含 3,168 个问题、45,632 次执行。在 1,632 个留出问题上，历史校准将 Brier 误差降低 16.2%，但测试的历史估计器仅能解释搜索增益方差（标准差 21.4 个百分点）的不到 2%。
正文
Abstract:AI agents must decide when another search is worth its cost. Answer-success probabilities capture question difficulty, while this decision requires estimating the additional benefit of search. We introduce SearchHorizonBench, a benchmark for learning and evaluating search value from execution experience. Paired continuations and repeated executions separate expected search gain from generation noise and connect prediction quality to decision utility. The resource contains 45,632 executions across 3,168 questions. On 1,632 held-out questions, historical calibration reduces Brier error by 16.2%, with the improvement concentrated in success-level information. Search gains have a standard deviation of 21.4 percentage points, and tested historical estimators explain less than 2% of this variance in point estimates. Matched-configuration learning curves probe this distinction across training scales. At the reference token price, free selection observations yield a 2.49-point utility opportunity over the best constant action, driven primarily by gain information. Reusable histories, frozen splits, strong baselines and scoring tools provide a common foundation for developing and evaluating value-aware search agents.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.34714 [cs.AI] |
| (or arXiv:2610.34714v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.34714 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xueqi Li [view email]
[v1]
Mon, 26 Oct 2026 02:04:38 UTC (275 KB)
来源：arXiv cs.AI · arxiv.org