arXiv cs.CL· Haram Choi·· 11 小时前AI 评分18
未筛选 NLI 人群中的单调性与标签一致性：一项复现研究
Selection Shapes the Boundary: A Replication of Monotonicity and Label Agreement in Unselected NLI Populations
AI 导读
一项预注册复现研究在 SNLI 和 MultiNLI 开发集上检验 ChaosNLI 的单调性边界，结果计划预测失败：全部七组对比的 Cliff's delta 均为正，非向上单调性条目的标签一致性略高而非略低，唯一显著的验证性对比方向与预期相反，且所有效应均远低于 0.10 的最小关注效应量。
正文 · 原文
Abstract:Prior work on human label variation (HLV) in natural language inference (NLI) has often relied on re-annotation resources that select items by disagreement level. An earlier study (Choi, 2026, arXiv:2607.15870) found that hypotheses containing non-upward monotonicity operators showed somewhat higher label entropy in ChaosNLI (Cliff's delta = -0.284), a difference that study reports as sensitive to tagger error. In its SNLI and MultiNLI portions, ChaosNLI re-annotates only items whose majority label carries exactly three of five votes. We replicate this boundary in the unselected populations that ChaosNLI was drawn from. We use the SNLI and MultiNLI development sets with the same operator tagger and a four-level ordinal agreement outcome, under a plan frozen in our working repository before analysis. The planned prediction fails. All seven contrasts return a positive Cliff's delta (non-upward items agree slightly more, not less), the only significant confirmatory contrast has the opposite sign to the planned direction, and every effect is far below our smallest effect size of interest (0.10). Robustness checks support the measurement: simulated tagger misclassification shrinks the effects rather than manufacturing them, and a manual re-tagging audit reaches four-class agreement of 0.875 on a fresh 200-item sample. We conclude that the earlier negative boundary is plausibly a structure conditional on low-agreement selection rather than a population-level property, and that HLV structure claims built on selected re-annotation resources should state their selection conditional explicitly.
| Comments: | 11 pages, 2 figures, 9 tables. Replication with a plan frozen before analysis. Code, data, and audit trail: this https URL |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.19231 [cs.CL] |
| (or arXiv:2607.19231v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.19231 arXiv-issued DOI via DataCite |
Submission history
From: Haram Choi [view email]
[v1]
Tue, 21 Jul 2026 16:03:05 UTC (46 KB)
[v2]
Sat, 31 Oct 2026 11:23:17 UTC (47 KB)
来源：arXiv cs.CL · arxiv.org