arXiv cs.AI· Yiding Ma, Chengyun Ruan, Kaibo Huang, Jin Tan, Zhongliang Yang, Linna Zhou·· 3 天前AI 评分26
OracleProto：通过知识截止与时间掩码对 LLM 预测能力进行基准测试的可复现框架
OracleProto: A Reproducible Framework for Benchmarking LLM Forecasting via Knowledge Cutoffs and Temporal Masking
AI 导读
OracleProto 是一个在受控时间掩码环境下评估 LLM 预测能力的可复现框架，通过模型知识截止约束和时间掩码构建时间沙盒，并借助泄漏检测审计信息边界。该框架在 300 道 FutureX-Past 衍生问题上对 27 组模型配置进行了 24,300 次预测试验，评估预测质量、采样稳定性和成本效率，检索结果残余泄漏率仅 1.37%。代码已开源。
正文 · 原文
Abstract:Forecasting is a composite capability encompassing information gathering, evidence integration, situational assessment, and decision-making. Large language models (LLMs) combine reasoning and tool use, offering new avenues for automated forecasting and decision support. Yet LLM forecasting evaluation faces a structural challenge: live benchmarks support prospective assessment, but limited question reuse and changing information conditions make results difficult to reproduce and compare; retrospective benchmarks offer reproducibility but risk information leakage. We introduce OracleProto, a reproducible framework for evaluating LLM forecasting in a controlled, temporally masked environment. Through model knowledge cutoff constraints and temporal masking, OracleProto constructs a temporal sandbox for real-world web retrieval, audits information boundaries through leakage detection, and reconstructs resolved events as forecasting instances. Discrete answer normalization and hierarchical scoring enable unified evaluation. We evaluate 27 model-configuration pairs on 300 FutureX-Past-derived questions over 24,300 prediction trials, assessing forecasting quality, sampling stability, and cost efficiency under controlled information boundaries. An audit finds only 1.37% residual leakage in retrieval results after temporal masking and leakage filtering. OracleProto retains resolved forecasting tasks as reusable evaluation resources under explicit temporal boundaries, supporting continual accumulation of forecasting datasets and systematic evaluation of forecasting capabilities. Code is available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.03762 [cs.AI] |
| (or arXiv:2605.03762v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.03762 arXiv-issued DOI via DataCite |
Submission history
From: Yiding Ma [view email]
[v1]
Tue, 5 May 2026 13:50:50 UTC (1,216 KB)
[v2]
Tue, 13 Oct 2026 17:06:43 UTC (1,797 KB)
来源：arXiv cs.AI · arxiv.org