跳到正文
arXiv cs.CL· Yicheng Tao, Yuante Li, Zipei Fan·· 23 小时前AI 评分42

ForecastGym:面向预测智能体的实时、可控、可回放基准

ForecastGym: A Live, Controlled, and Replayable Benchmark for Forecasting Agents

AI 导读

ForecastGym 是一个面向 LLM 预测智能体的实时、可控、可回放基准,每两天记录四个商业搜索 API 对开放预测市场问题的返回结果,结算后离线回放冻结证据。首个版本覆盖 150 个问题、614 个证据状态和 131k 条搜索结果。在冻结证据上,所有受测 LLM 的预测准确率均落后于市场;更换搜索 API 可使模型 Brier 分数变化达 0.043,超过模型间差异。

正文 · 原文

View PDF HTML (experimental)

Abstract:Agents built on large language models (LLMs) increasingly forecast real-world events by searching the Web and reasoning over what they find. How well such an agent forecasts depends on both the retrieved evidence and the model, making control over that evidence essential for evaluation. Live evaluations can mix retrieval with reasoning when each system searches on its own, while retrospective search can leak later information through date filters or today's ranking of historical pages. Comparing models, search sources, and memory strategies requires both fixed evidence and a record of how it evolved before resolution. We introduce ForecastGym, which records what four commercial web search APIs return for open prediction-market questions every two days and replays this frozen evidence offline once outcomes are known, comparing models, memory, and captured search sources under a fixed evidence-acquisition policy. Its first release covers 150 questions, 614 evidence states, one per question and collection time, and 131k search results. Rerunning the same queries after settlement returns results that all but state the outcome for up to 92% of the questions, despite the tested date filters. On the frozen evidence, all tested LLMs trail the market in forecasting accuracy. Our diagnostic analysis suggests that missed public information contributes more to the gap in high-gap states than misreading does. Switching the search API changes a model's Brier score by up to 0.043, more than the spread across models, while persistent notes score similarly to rereading all evidence at half the input tokens. The evidence a forecaster sees can matter as much as the model that reads it.
Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as: arXiv:2611.23045 [cs.AI]
  (or arXiv:2611.23045v1 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2611.23045

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yicheng Tao [view email]
[v1] Tue, 17 Nov 2026 16:44:38 UTC (467 KB)

来源:arXiv cs.CL · arxiv.org