跳到正文
arXiv cs.AI· Jinhyung Bae·· 10 小时前AI 评分31

LexTemporis 用韩国上市公司披露记录评测 LLM 编写的可执行法律规则

Grading Executable Legal Rules Against Independent Records: Transitional Provisions and Korean SME Status

AI 导读

LexTemporis 提出用真实披露记录而非评分者自读法条来评测 LLM 编写的可执行法律规则,测试集为韩国上市公司 10,901 个公司-年份的 SME 状态披露。

正文 · 原文

View PDF HTML (experimental)

Abstract:Laws are increasingly turned into programs that decide cases, such as whether a firm was a small or medium enterprise (SME) on a given date, and large language models (LLMs) now write them. The programs are usually graded against answers written by someone who read the same law, so a program can score well by sharing the grader's reading rather than by getting the law right. LexTemporis grades them instead against records of what actually happened: 10,901 firm-years of SME status disclosed by Korean listed companies, with every version of the governing rules. An appendix adds comparison cases from US income tax and German hospital minimum volumes. The date changes outcomes mainly through provisions outside the main statutory text: one transitional provision raises agreement with the disclosures from 83-86% to 89-90%. The disclosures also test the author's reading of a contested provision, favouring it in 130 of 189 firm-years where readings diverge. Four LLMs wrote the rules, given different selections of the law. A pre-registered comparison finds no condition better across all four models. In exploratory analyses, supplying the right provisions looks better graded against the author's reading than against the records: the advantage shrinks for three models (for one, to between a quarter and three-fifths of its size), and per firm-year it is several times larger where the author's reading is wrong, partly by construction, since there reproducing that reading more faithfully must lose. Grading legal programs against the grader's own reading can overstate how well they work.
Comments: 51 pages, 2 figures (plus 1 in the appendix). Submitted to Artificial Intelligence and Law. Code and data: this https URL
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.24055 [cs.CL]
  (or arXiv:2611.24055v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.24055

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Jinhyung Bae [view email]
[v1] Wed, 18 Nov 2026 08:46:52 UTC (367 KB)

来源:arXiv cs.AI · arxiv.org