arXiv cs.CL· Sebastian Nagl·· 3 天前AI 评分24
SoK：开放文本法律答案的评分，从法学院考试到可信的评分标准评审器
SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judges
AI 导读
这篇 SoK 系统梳理了开放文本法律答案的评估方法，从教育测量与考试法出发提出有效性、可靠性、理由、可辩护替代方案得分、抗博弈与人工监督等八项要求，并对比了从非结构化人工评分到带实例化评分标准的 LLM 评审器、成对比较等八种评估范式。
正文 · 原文
Abstract:Law students are examined through open-text answers: case solutions, essays and memos that apply the law to a set of facts. Such answers are expensive to grade, human graders disagree widely on them, and large language models (LLMs) now offer to grade them at scale, both in teaching and in the benchmarks that test legal AI. This paper systematizes how open-text legal answers can be evaluated. We derive eight desiderata from educational measurement and from the law that governs exams, among them validity, reliability, reasons, credit for defensible alternative solutions, robustness to gaming and human oversight. We then compare eight evaluation paradigms against them, from unstructured human grading to LLM judges with instance-specific rubrics and pairwise comparison. Three findings stand out. A rubric written for the task is the strongest lever for agreement between an LLM judge and human graders, but the reported figures answer different questions and rarely come with the human agreement they must be read against. The central validation gap lies where examination law is most demanding: defensible alternative solutions, follow-on errors, and the examiner's correction of model judgments. And rubric-centred grading is the most auditable family, because it makes the standard, its application and the final decision checkable. From this we derive the trustworthy rubric judge: an instance rubric that credits arguments and alternative paths,evidence checked in code, probes, repeated runs, calibration against independent human grades,logging, and an examiner who reads every script. We state what each safeguard rests on, what remains to be tested, and the uses the evidence supports today.
| Comments: | preprint |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19992 [cs.CL] |
| (or arXiv:2610.19992v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19992 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sebastian Nagl [view email]
[v1]
Wed, 14 Oct 2026 11:09:44 UTC (342 KB)
来源：arXiv cs.CL · arxiv.org