arXiv cs.CL· Shimaa Ibrahim, Mabrouka Bessghaier, Md. Rafiul Biswas, Insaf Elouard, Wajdi Zaghouani·· 11 小时前AI 评分26
超越关键词：阿拉伯语焦虑相关话语的人工标注基准发布
Beyond the Keyword: A Human-Annotated Benchmark for Arabic Anxiety-Related Discourse
AI 导读
研究者发布了一个包含 1000 条阿拉伯语 Twitter/X 帖子的基准，这些帖子以关键词 qalaq（焦虑/担忧）检索并按六种话语功能标注。两名母语标注者独立标注的原始一致率为 51.1%（Cohen's κ = 0.378），复核全部 489 处分歧和 511 处一致后，有 120 条（23.5%）一致标签被修改。
正文 · 原文
Abstract:Keyword retrieval returns expressions of personal anxiety together with many other uses of anxiety-related language. We present a benchmark of 1,000 Arabic Twitter/X posts retrieved with the keyword qalaq ("anxiety/worry") and annotated for six discourse functions. Two native Arabic-speaking annotators labeled every post independently, reaching 51.1% raw agreement (Cohen's $\kappa = 0.378$). We then reviewed all 489 disagreements and all 511 agreements under the same guidelines. The review changed 120 of the agreed labels (23.5%), which shows that agreements need auditing as well as disagreements. The final labels form a consensus reference; on a random sample of 200 posts, an additional annotator agreed with it on 60.5% of posts ($\kappa = 0.517$). Against this reference, Fanar-C-2-27B reaches 79.8% accuracy and macro-F1 0.784 ($\kappa = 0.749$). Its errors concentrate in the residual category (41.1% of errors; precision 0.905, recall 0.447), whose definition is internally inconsistent in the executed prompt, so the low recall cannot be attributed to the model alone. Sociopolitical Anxiety (27.1%) outnumbers Personal Emotional Expression (22.0%); these are assignment proportions in the sample, not estimates of prevalence or retrieval precision. We release the guidelines, a residual-subtype scheme, a fixed data split, label provenance, and evaluation code.
| Comments: | 13 pages, 1 figure, 6 tables |
| Subjects: | Computation and Language (cs.CL); Social and Information Networks (cs.SI) |
| Cite as: | arXiv:2611.07949 [cs.CL] |
| (or arXiv:2611.07949v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.07949 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shimaa Ibrahim [view email]
[v1]
Thu, 5 Nov 2026 12:45:49 UTC (688 KB)
来源：arXiv cs.CL · arxiv.org