arXiv cs.CL· Hao Lu, Weixing Zhu, Ruiyang Li·· 23 小时前AI 评分29
契约显式评估:从审计学习型阅读器到范围化证书
Contract-Explicit Evaluation: From Auditing Learned Readers to Scoped Certificates
AI 导读
研究提出契约显式评估协议,通过 Declare、Measure、Scope、Certify、Hedge 五步操作化 Kendall 秩相关的经典一致性分解,为模型比较划定可认证区域并给出无分布假设的普查证书。
正文 · 原文
Abstract:Evaluation readings of document-level relation extraction (DocRE) systems depend jointly on the model and the evaluation contract: gold version, scoring function, threshold. Rankings reverse across contracts, yet no existing method determines which comparisons survive the reversal. We address this gap with contract-explicit evaluation: a preregistered five-step protocol (Declare, Measure, Scope, Certify, Hedge), operationalizing the classical concordance decomposition of Kendall's rank correlation, that scopes certifiable regions and delivers distribution-free census certificates across two ecosystems. In the DocRE audit (44 training runs of one in-house architecture family), the scoring dimension scopes to a census certificate (three flagged instances, Clopper--Pearson upper bound 0.146); the confirmatory record includes registered negatives, and the audit prices its own simplification finding---the calibration dial's entire margin is one withdrawn commit. In an LLM-judge census (17 judges, 49,938 battles), 12.25% of leaderboard comparisons require hedging at the pre-declared split criterion (bound 0.124; panel-robust 0.143), with an exploratory two-tier reading certifying 69% at a bound \le 0.072 and a validity decomposition against released human labels (65% of contested battles are judge error, not ambiguity). The cross-model certificate is withdrawn on real confidence scores; the failure spectrum maps the boundary. The protocol, map, ledger, and per-instance artifacts are released.
| Comments: | 50 pages, 4 figures. Supplementary material (Online Resource 1, 14 pages) included as ancillary file. Artifact repository: this https URL. Submitted to Applied Intelligence |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.22665 [cs.CL] |
| (or arXiv:2611.22665v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22665 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hao Lu [view email]
[v1]
Tue, 17 Nov 2026 12:22:27 UTC (225 KB)
来源:arXiv cs.CL · arxiv.org