arXiv cs.AI· Xu Liu, Hanshu Yan, Taha Aksu, Quang Pham, Zhongzheng Qiao, Jun Hao Liew, Silvio Savarese, Doyen Sahoo, Junnan Li·· 3 天前AI 评分24
GIFT-Eval-CTX：面向上下文预测模型评测的基准
GIFT-Eval-CTX: A Benchmark For Contextual Forecasting Model Evaluation
AI 导读
研究者推出 GIFT-Eval-CTX 基准，用于检验预测方法能否把已验证的未来信息转化为准确的数值轨迹。该基准基于 GIFT-Eval 构建，覆盖 20 个数据集、10,000 个实例，每个实例提供由 LLM 发现与确定性代码校验的智能体流水线生成的 Macro 与 Micro 结构化效应。
正文 · 原文
Abstract:Contextual forecasting seeks to improve time series forecasting by incorporating forward-looking information, such as planned events, external conditions, and expert expectations. However, reliable evaluation remains difficult: existing benchmarks often have limited domain coverage, lack standardized history-future splits, or provide text that is not guaranteed to inform the future. We introduce GIFT-Eval-CTX, a benchmark that isolates whether forecasting methods can translate verified future information into accurate numerical trajectories. Built upon GIFT-Eval, GIFT-Eval-CTX contains 10,000 instances across 20 datasets. Each instance provides structured effects derived from the realized future: Macro effects describe horizon-wide behavior, while Micro effects specify localized changes. We construct these annotations with an agentic pipeline combining LLM-based discovery and deterministic code validation. We benchmark TSFMs, frontier LLMs, and tool-using agentic systems. The results show that structured effects reduce aggregate relative MASE by 40.7% for GPT-5.6 Sol and 29.6% for Claude Opus 4.8 relative to history-only prompting. The evaluated MoiraiAgent system achieves the best overall result. We further analyze the contributions of structured effects and agentic systems to performance, and identify opportunities for future research.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.17794 [cs.LG] |
| (or arXiv:2610.17794v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.17794 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xu Liu [view email]
[v1]
Tue, 13 Oct 2026 06:21:50 UTC (270 KB)
来源：arXiv cs.AI · arxiv.org