arXiv cs.CL· Yinuo Xu, David Jurgens·· 12 小时前AI 评分26
超越共识：NLP 中标注者分歧的视角主义建模与评估综述
Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP
AI 导读
一篇被 EMNLP 2026 接收的综述系统梳理了 NLP 中标注者分歧感知方法，提出跨数据、任务与标注者三类因素的分歧来源分类体系，并用统一的预测目标与池化结构框架归纳建模路径，指出研究正从共识学习转向显式建模分歧及标注者间结构关系。文章还回顾了预测性能与标注者行为两类评估指标，并指出多数公平性评估仍停留在描述性层面而非规范性判断。
正文 · 原文
Abstract:Annotator disagreement is widespread in NLP, particularly for subjective and ambiguous tasks such as toxicity detection and stance analysis. While early approaches treated disagreement as noise to be removed, recent work increasingly models it as a meaningful signal reflecting variation in interpretation and perspective. This survey provides a unified view of disagreement-aware NLP methods. We first present a domain-agnostic taxonomy of the sources of disagreement spanning data, task, and annotator factors. We then synthesize modeling approaches using a common framework defined by prediction targets and pooling structure, highlighting a shift from consensus learning toward explicitly modeling disagreement, and toward capturing structured relationships among annotators. We review evaluation metrics for both predictive performance and annotator behavior, and noting that most fairness evaluations remain descriptive rather than normative. We conclude by identifying open challenges and future directions, including integrating multiple sources of variation, developing disagreement-aware interpretability frameworks, and grappling with the practical tradeoffs of perspectivist modeling.
| Comments: | Accepted to EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2601.09065 [cs.CL] |
| (or arXiv:2601.09065v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2601.09065 arXiv-issued DOI via DataCite |
Submission history
From: Yinuo Xu [view email]
[v1]
Wed, 14 Jan 2026 01:26:29 UTC (263 KB)
[v2]
Sat, 17 Jan 2026 17:58:09 UTC (268 KB)
[v3]
Fri, 16 Oct 2026 16:39:26 UTC (270 KB)
来源：arXiv cs.CL · arxiv.org