arXiv cs.CL· Yevhen Kostiuk, Kenneth Enevoldsen, Peter Bjerregaard Vahlstrup, M\'arton Kardos, Kristoffer Nielbo·· 13 小时前AI 评分20
用自然场景下的解决方案匹配衡量 LLM 社会规范对齐
Naturalistic measure of social norms alignment
AI 导读
研究者提出一套在自然、自由形式场景下通过解决方案匹配衡量社会规范对齐的框架，可测量 LLM 与人类、LLM 与 LLM、人与人之间对困境回答的一致程度。该框架引入 stated 与 explicit agreement accuracy 两项指标，并构建了含 3k 个丹麦语非平凡社会困境的数据集，每个困境的参考解决方案由三位评审给出。
正文 · 原文
Abstract:Social norms reflect shared expectations on acceptable behavior. Measuring social norms alignment remains challenging, with existing approaches typically relying on artificial closed-form evaluations such as multiple-choice questionnaires or measuring agreement with predefined statements. In the context of this work, social norms alignment refers to measuring an agreement between solutions with respect to the social problem or dilemma. We propose a framework for measuring social norm alignment in naturalistic, free-form settings through solution matching. The framework enables us to measure alignment between any two dilemma responses e.g., LLMs to a human, LLMs to LLMs, or human to human. We introduce two metrics: stated and explicit agreement accuracy, and construct a dataset of 3k non-trivial social dilemmas in Danish. All dilemmas are assigned reference solutions derived from three panelists, who serve as culturally grounded judges. We evaluate the agreement of several LLMs and human responses in an interaction setup that resembles natural user-model conversations. Our results show that the proposed metrics produce consistent model rankings and reveal variation in agreement across different types of dilemmas, with higher agreement observed for topics such as neighbor conflicts and shared living situations. Overall, our work introduces a dataset and evaluation framework for studying culturally grounded social reasoning in naturalistic open-ended conversations.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.23420 [cs.CL] |
| (or arXiv:2605.23420v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.23420 arXiv-issued DOI via DataCite |
Submission history
From: Yevhen Kostiuk [view email]
[v1]
Fri, 22 May 2026 09:29:18 UTC (13,916 KB)
[v2]
Mon, 26 Oct 2026 10:05:32 UTC (7,057 KB)
来源：arXiv cs.CL · arxiv.org