arXiv cs.CY· M. Mikail Demir·· 13 小时前AI 评分42
该提交还是弃权？System One 决策模型、LLM 与缺失的法律依据
Commit or Abstain? System One Decision Models, LLMs, and Missing Legal Authority
AI 导读
研究将 CaseHOLD 扩展出弃权条件（5,314 题中 1,670 题替换正确判决），对比四个 System One 决策模型与六个生成式 LLM：无正确选项时生成式 LLM 仍有 38-86% 强行选择，最谨慎的决策模型为 30%。
正文 · 原文
Abstract:Given the enormous number of claims to be checked, verifiable legal AI requires scalable, cost-effective models that answer the practical needs of law. System One decision models are built on large language models but, instead of generating text, return a calibrated probability for each option. This suits the verification of legal authority, where the question is closed: a given holding, or none of them. We test whether such models abstain when the correct authority is missing. We extend CaseHOLD with an abstention condition in which the correct holding is replaced in 1,670 of 5,314 items, and compare four decision models with six generative LLMs. When no option is correct, generative LLMs still pick one in 38-86% of items; the most cautious decision model does so in 30%. On standard CaseHOLD, a decision model matches the best generative LLM (88.1% vs. 88.2%) at 1/55 of the cost. Accepting that LLM's answer only when a decision model agrees halves its wrong citations, and distilling a decision model's probabilities into Qwen3-8B, without labels, yields a well-calibrated local model. Legal AI should be judged by its wrong citations, not by accuracy alone.
| Comments: | 16 pages, 4 figures, 4 tables. Under review. Code, data, and model outputs: this https URL |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY) |
| Cite as: | arXiv:2610.29961 [cs.AI] |
| (or arXiv:2610.29961v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.29961 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: M. Mikail Demir [view email]
[v1]
Wed, 21 Oct 2026 16:17:49 UTC (1,027 KB)
来源：arXiv cs.CY · arxiv.org