arXiv cs.CL· Philipp E. Glass, Alina Miron·· 9 天前AI 评分26
CompOrca：为 OpenOrca 全量 423 万条指令微调数据打合规标签
CompOrca: Corpus-Scale Compliance Labelling of Instruction-Tuning Data
AI 导读
CompOrca 为 OpenOrca 语料全部 4,233,923 条样本提供合规标签，由开源权重模型 LongCat-2.0（1.6T 参数）进行五轮判定，并划分为一致合规（94.75%）、一致不合规（1.28%）与非一致行（3.97%）。
正文 · 原文
Abstract:Studying how fine-tuning shapes refusal and noncompliance behaviour requires knowing which training examples refuse or otherwise fail to fulfil the request. Existing annotations cover evaluation sets, which are far smaller than training corpora. We present CompOrca, compliance labels for all 4,233,923 examples of the OpenOrca corpus. Every example was classified as compliant or noncompliant by five passes of an open-weight LLM judge (LongCat-2.0, 1.6T parameters). The corpus is released as unanimous compliance (94.75%), unanimous noncompliance (1.28%), and nonunanimous rows (3.97%), with the raw vote counts. A single pass flags 2.7-3.2% of the corpus as noncompliant, while only 1.28% is flagged by all five, so the most ambiguous rows can be filtered out. Against 450 human-annotated examples (150 annotated twice; human-human $\kappa=0.93$), the unanimous compliance and noncompliance labels are 97.3% and 86.7% precise. The noncompliance label is a high-precision subset of the corpus's noncompliance. Published refusal-detection methods recall between 0.4% and 94.1% of human-labelled noncompliance. We release the full corpus with its per-row labels and vote counts at this https URL
| Comments: | Accepted to PlurVA-LLM Workshop @ AACL-IJCNLP 2026. Dataset available on HuggingFace |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.37807 [cs.CL] |
| (or arXiv:2609.37807v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.37807 arXiv-issued DOI via DataCite |
Submission history
From: Philipp E. Glass [view email]
[v1]
Tue, 29 Sep 2026 15:25:07 UTC (20 KB)
[v2]
Thu, 8 Oct 2026 00:51:13 UTC (135 KB)
来源：arXiv cs.CL · arxiv.org