arXiv cs.CL· Rilton Franzone, Magnus Sesodia, Alina Petrova, John Armour, Thomas Lukasiewicz, Oana-Maria Camburu, Puneet K. Dokania, Philip Torr, Christian Schroeder de Witt, Fabio J. Fehr·· 13 小时前AI 评分29
AnnoCaseLaw：美国民事过失法律标注基准发布
AnnoCaseLaw: Legal Annotation Benchmark of Civil Negligence
AI 导读
AnnoCaseLaw 基准包含 471 份美国民事过失法院意见书和超过 12,000 条专家标注，覆盖五类标注类别，用于评估语言模型复现专家法律标注的能力。
正文 · 原文
Abstract:Language models offer a way to scale costly, labour-intensive expert annotation, but their ability to reproduce expert judgements remains uncertain. This is particularly challenging in law, where annotators must distinguish the reasoning supporting judicial decisions from other legally relevant discussion. We present AnnoCaseLaw as a densely annotated benchmark comprising $471$ U.S. civil-negligence court opinions with more than $12,000$ expert highlights across five annotation categories. We evaluate proprietary frontier, open-weight, legal-specialised, and decision models on their ability to reproduce these annotations. Models perform better on Facts and Outcome than on Application of Law to Facts, Procedural History, and Relevant Precedents. On Application of Law to Facts, proprietary frontier models achieve the strongest balance of reasoning coverage and precision, while the open-weight model Gemma 4 achieves comparable Macro-F1 to proprietary frontier models at a fraction of the estimated inference cost. Our findings highlight the importance of evaluating annotation coverage and cost alongside aggregate scores when assessing the feasibility of automated legal annotation.
| Comments: | Revised version. AnnoCaseLaw is refocused from legal judgment prediction to expert legal annotation. The corpus and expert annotations are retained, while the annotation task, model evaluations, experiments, and analyses are revised and extended. The author list has been updated to reflect the contributions to this version |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2503.00128 [cs.CL] |
| (or arXiv:2503.00128v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2503.00128 arXiv-issued DOI via DataCite |
Submission history
From: Fabio James Fehr [view email]
[v1]
Fri, 28 Feb 2025 19:14:48 UTC (598 KB)
[v2]
Wed, 21 Oct 2026 15:19:43 UTC (1,118 KB)
来源：arXiv cs.CL · arxiv.org