arXiv cs.CL· Denglin Jiang (New York University), Annie Wang (New York University)·· 12 小时前AI 评分31
社会世界模型评估缺口：113 项研究审计发现仅 2 项直接检验对他人反应的预测
The Social World Model Evaluation Gap
AI 导读
一项审计 1999 至 2026 年 113 项研究的分析发现，仅 9 项主要评估测量了真实的人对系统或实验者所选动作的反应，其中只有 2 项还将显式反应预测与该反应作对比。对 25 项语言模型工作的全文检查未发现满足这些条件的实验，其余评估测试的是当前状态推断、任务成功或基于记录动作的预测。作者提出一个 1-step 谈判实验来区分这些问题，并说明结论仅描述所审计样本。
正文
Abstract:A negotiation assistant can reach a good deal without accurately predicting the other person's response. We ask whether evaluations of social world models test that prediction directly. We audit 113 works published from 1999 to 2026. 9 primary evaluations measure a real person's response to an action selected by the system or experimenter. Only 2 also compare an explicit response forecast with that response. Their predictors use information available when the action is chosen, and the evaluated responses were excluded from fitting; both randomize the intervention. A separate full-text check of 25 language-model works finds no experiment meeting these conditions. The other evaluations test current-state inference, task success, or prediction from recorded actions. We propose a 1-step negotiation experiment that separates these questions. The findings describe the audited sample, not the prevalence of this evaluation gap across the field.
| Comments: | Accepted at the NeurIPS 2026 Workshop on Foundation Models for Temporal Systems: From Forecasting to World Modeling (FMTS). 4 pages of main text plus references and appendix. Code and data: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.33014 [cs.CL] |
| (or arXiv:2610.33014v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33014 arXiv-issued DOI via DataCite |
Submission history
From: Denglin Jiang [view email]
[v1]
Thu, 8 Oct 2026 04:18:33 UTC (110 KB)
来源：arXiv cs.CL · arxiv.org