ALMANAC：面向智能体协作的人类协作数据集，含动作级心智模型标注
Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaboration
研究者发布 ALMANAC 数据集，基于社会科学经典双人 Map Task 构建，包含 2,987 条协作动作，每条均配有理论驱动的心智模型标注，记录参与者自我推理、感知到的伙伴意图与团队目标。团队在预测人类下一轮行为与心智模型任务上基准测试了六个 LLM，结果显示该数据集可用于评估模型模拟人类协作行为并推断其心智模型的能力。该工作已被 EMNLP 2026 接收为 Findings。
Authors:Jiaju Chen, Yuxuan Lu, Jiayi Su, Chaoran Chen, Songlin Xiao, Zheng Zhang, Yun Wang, Yunyao Li, Jian Zhao, Tongshuang Wu, Toby Jia-Jun Li, Dakuo Wang, Bingsheng Yao
Abstract:Recent advances in LLM agents have enabled complex cognitive capabilities, such as multi-step reasoning, planning, and tool use, that increasingly position these agents as collaborative partners in human workflows. Effective collaboration, however, requires collaborators to continuously maintain and align mental models of their own reasoning, partners' intentions, and shared goals during the collaborative process. Today's agents rarely develop such capabilities since they are primarily optimized for task completion rather than for maintaining the underlying mental models required for collaborative processes. Moreover, the community lacks authentic human collaboration data with action-level mental model annotations that could guide agents toward collaborative competence. To bridge this gap, we present ALMANAC, a dataset of Action-Level Mental model ANnotations for Agent Collaboration built from the Map Task, a classic dyadic routing task from social science. ALMANAC contains 2,987 collaboration actions, each paired with theory-informed mental model annotations that record the participants' self-reasoning, perceived partner intent, and perceived team goal. We benchmark six LLMs on predicting humans' next-turn behavior and mental models. Our results demonstrate ALMANAC's utility in evaluating models' ability to simulate human collaborative behaviors and infer their underlying mental models. The project page is available at this https URL.
| Comments: | Accepted as Findings of EMNLP 2026 Conference |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| MSC classes: | 68T50 |
| Cite as: | arXiv:2606.06388 [cs.AI] |
| (or arXiv:2606.06388v3 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.06388 arXiv-issued DOI via DataCite |
Submission history
From: Jiaju Chen [view email]
[v1]
Thu, 4 Jun 2026 16:56:12 UTC (2,396 KB)
[v2]
Sat, 6 Jun 2026 00:32:51 UTC (2,396 KB)
[v3]
Sun, 1 Nov 2026 03:39:31 UTC (2,500 KB)
来源：arXiv cs.CL · arxiv.org