Multi-Label Agreement：面向多标签标注者间信度的灵活框架

View PDF HTML (experimental)

Abstract:Reliable evaluation of inter-annotator agreement (IAA) is important for assessing the consistency of annotation processes. While established IAA metrics such as Cohen's kappa, Fleiss' kappa, and Krippendorff's alpha are commonly applied in single-label settings, multi-label tasks require additional decisions about how full label sets, joint selections, and joint omissions should be compared.
In this work, we present Multi-Label Agreement (MLA), a flexible framework for evaluating agreement in multi-label annotation tasks. MLA preserves the full binary label vectors assigned to each item and uses average pairwise Hamming agreement, thereby accounting for both joint selections and joint omissions. Its pooled-marginal form is not a mathematically independent coefficient: it is a chance-disagreement-weighted average of label-wise Fleiss/Scott-type coefficients and is closely related to Krippendorff's alpha with normalized Hamming distance. The framework supports pooled-marginal, Gwet AC1, and fixed-baseline normalizations, yields item-level, label-level, and corpus-level results, and includes a conditional minimum-agreement diagnostic. Through a worked example and Monte Carlo simulations, we show how the resulting values depend on label prevalence, the declared label inventory, and the selected model of expected agreement. We also provide a Python implementation. MLA thus offers a transparent and interpretable framework for multi-label agreement analysis without claiming that one normalization is universally preferable.

Subjects:	Computation and Language (cs.CL)
ACM classes:	G.3; I.2.6; I.2.7
Cite as:	arXiv:2610.37292 [cs.CL]
	(or arXiv:2610.37292v1 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2610.37292 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Tim Menzner [view email]
[v1] Tue, 27 Oct 2026 14:15:12 UTC (209 KB)