arXiv cs.CL· Yunhua Pei, Jingyu Hu, Yiwei Shi, Hongnan Ma, Weiru Liu, John Cartlidge·· 10 小时前AI 评分34
StakeBench：基于市场承诺评估语言理解的新基准
StakeBench: Evaluating Language Understanding Grounded in Market Commitment
AI 导读
StakeBench 是一个基于市场承诺的语言理解评测框架，将 Polymarket 和 Manifold 上 2,261 个已结算市场的 560,876 条评论与持仓、交易动作和市场赔率记录关联，用可观测市场行为替代人工标注。
正文 · 原文
Abstract:Existing financial NLP benchmarks often rely on labels supplied by outside observers, measuring how language is perceived rather than what speakers have committed to in the market. We introduce StakeBench, an evaluation framework for language understanding grounded in market commitment. StakeBench links 560,876 comments from 2,261 resolved markets to verified position, action, and market-odds records across Polymarket and Manifold. Supervision is derived from observable market behavior. Position sides, post-comment trading actions, and market-odds trajectories replace human annotation. Four diagnostic tasks test whether models detect market commitment, identify the revealed side, anticipate future action, and perform collective odds projection. Three commitment-aware metrics measure alignment with revealed preferences rather than perceived sentiment. Validity audits and explicit interpretation boundaries help distinguish observable commitment signals from latent belief and causal market-odds impact. Across 15 LLMs and 18 topics and platform settings, models partially recover position-side signals, with Directed Accuracy from 0.506 to 0.599, but show structural failures on later tasks. Ten of the fifteen models collapse to one or two action labels in future action anticipation, and no model consistently improves on the naive odds-direction baseline in collective odds projection. Model scale is not correlated with performance, finance-domain tuning does not improve revealed-side identification, and platform incentives strongly shape higher-order results. StakeBench is packaged with evaluation code and dataset under CC-BY 4.0.
| Comments: | 28 pages, 2 figures, 20 tables. Accepted author manuscript, NeurIPS 2026 (Evaluations and Datasets Track). Dataset and evaluation code included |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); General Finance (q-fin.GN) |
| Cite as: | arXiv:2605.26074 [cs.CL] |
| (or arXiv:2605.26074v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.26074 arXiv-issued DOI via DataCite |
Submission history
From: John Cartlidge [view email]
[v1]
Mon, 25 May 2026 17:38:30 UTC (740 KB)
[v2]
Tue, 10 Nov 2026 10:59:17 UTC (758 KB)
来源：arXiv cs.CL · arxiv.org