arXiv cs.CY· Alaa Bazaid, Arkaitz Zubiaga·· 23 小时前AI 评分31
MuDAS 数据集揭示:多数标签掩盖了标注者分歧与模型错误——多方言阿拉伯语性别歧视检测研究
What Majority Labels Hide: Annotator Disagreement and Model Errors in Multi-Dialect Arabic Sexism Detection
AI 导读
研究者构建 MuDAS 数据集,收录沙特、埃及、摩洛哥三种阿拉伯语方言的 YouTube 评论,每条由 3 名母语者独立标注并保留个体判断,用于评估阿拉伯语编码器、GPT-5.6 Terra 及四个阿拉伯语大模型。
正文 · 原文
Abstract:Judgements of sexism are subjective and may vary across communities, yet sexism detection datasets usually reduce annotator judgements to a single majority label, obscuring disagreement and how models behave on contested cases. We introduce MuDAS, a dataset of YouTube comments in Saudi, Egyptian and Moroccan Arabic, each labelled by three native speakers of its dialect, with the individual annotations preserved. We use it to evaluate Arabic encoders fine-tuned with hard-label, soft-label and perspectivist objectives, GPT-5.6 Terra, and four Arabic-centric large language models. Model errors are more frequent on comments with annotator disagreement. Despite relatively close Macro-F1 scores, GPT-5.6 Terra and the fine-tuned encoders exhibit contrasting error patterns: GPT-5.6 Terra, even with the annotators' definition of sexism, over-classifies gender-related content such as religious advice on modesty, whereas the encoders miss implicit and prescriptive sexism. The Arabic-centric models also over-predict sexism and remain below the encoders. Disagreement-aware learning preserves Macro-F1, but soft labels improve alignment with annotators on contested comments at the expense of unanimous ones, and neither paradigm reliably identifies contested comments. Lower performance on Egyptian Arabic, clearest for the large language models, is not fully explained by the examined measures of test-set composition and dialectness. Majority-label scores can therefore hide contrasting error patterns that matter for content moderation.
| Comments: | 36 pages, 6 figures. Submitted to Online Social Networks and Media. Dataset: this https URL . Content warning: contains examples of sexist and offensive language |
| Subjects: | Computation and Language (cs.CL); Computers and Society (cs.CY); Social and Information Networks (cs.SI) |
| Cite as: | arXiv:2611.24383 [cs.CL] |
| (or arXiv:2611.24383v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.24383 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Alaa Bazaid [view email]
[v1]
Wed, 18 Nov 2026 13:14:00 UTC (1,076 KB)
来源:arXiv cs.CY · arxiv.org