arXiv cs.CL· Xiaopeng Ke, Zhixin Wang, Xuebo Liu, Xinyu Ma, Xing Zheng, Bowen Song, Weiqiang Wang, Min Zhang·· 12 小时前AI 评分29
ANT-A Benchmark：LLM 标注能否匹配人类一致性？多智能体对比知识优化
Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive Knowledge
AI 导读
研究者提出 ANT-A Benchmark，覆盖 7 个跨领域、跨模态数据集，每个实例含 5 个独立人类标注，并用 Fleiss kappa 评估模型标注与人类标注分布的一致性。
正文 · 原文
Abstract:LLM-based annotation has advanced rapidly, yet existing benchmarks remain misaligned with real-world practice. Actual annotation often involves lengthy, task-specific guidelines, and different annotators may produce divergent but reasonable judgments, especially on boundary cases, where single-labeled dataset fail to reflect this disagreement. This gap also limits existing single-agent and workflow-based annotation systems, which often lack adaptability to complex, evolving guidelines. To bridge this gap, we introduce ANT-A Benchmark, a real-world multi-annotator benchmark spanning 7 datasets across domains and modalities, with task-specific guidelines and 5 independent human labels per instance. ANT-A enables a distribution-aware evaluation, using Fleiss kappa to assess how closely model annotations align with human labeling distributions. We further propose ANT-OR, a multi-agent annotation framework that refines prompts by distilling contrastive decision rules from positive and negative feedback. Experiments show that ANT-OR improves agreement@>=2 accuracy by 4.5 points over the strongest baseline and achieves the highest alignment with human annotator distributions as measured by Fleiss kappa within 2 points. Code and benchmark are available at this https URL.
| Comments: | Accepted to EMNLP 2026 Findings |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.32415 [cs.CL] |
| (or arXiv:2610.32415v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32415 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiaopeng Ke [view email]
[v1]
Fri, 23 Oct 2026 09:43:38 UTC (1,602 KB)
来源：arXiv cs.CL · arxiv.org