arXiv cs.CL· London Lowmanstone IV, Vaagishwaran Sivakumar, Anwesha Samaddar, Akul Mundada, Harnoor Kaur, Maria Gini·· 12 小时前AI 评分24
视角主义模型对哪些标注者预测更准？标注者一致性、稀疏性与性能分析
Who Do Perspectivist Models Predict Well? Annotator Agreement, Sparsity, and Performance
AI 导读
研究构建了迄今最大的带序数标注的视角主义 NLP 数据集集合，并评估多类视角主义算法后发现：算法对倾向于与他人一致的标注者预测更好，而标注者的响应数量与算法性能之间不存在统计显著的正相关。
正文 · 原文
Abstract:Successful completion of perspectivist natural language processing (NLP) annotation tasks requires algorithms to be aligned across diverse perspectives in order to predict how different humans will rate text. This work studies how the performance of such algorithms varies for annotators who behave differently than others, either by providing annotations that diverge from others' responses or by providing fewer annotations altogether. To aid with characterizing model performance, we curate the largest collection to date of perspectivist NLP datasets with ordinal annotations. After evaluating multiple classes of perspectivist algorithms across each of the datasets, we model the relationship between individual annotator idiosyncrasy and algorithm performance. In general, we find that algorithms perform better on annotators who tend to agree with others, but that annotator response rate does not have a statistically significant positive correlation with algorithm performance.
| Comments: | 24 pages, 9 figures, 5 tables |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.22567 [cs.CL] |
| (or arXiv:2610.22567v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22567 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: London Lowmanstone [view email]
[v1]
Thu, 15 Oct 2026 19:12:48 UTC (365 KB)
来源：arXiv cs.CL · arxiv.org