arXiv cs.CL· Pawitsapak Akarajaradwong, Sarana Nutanong, Chompakorn Chaksangchaichot·· 12 小时前AI 评分29
NitiBench+：覆盖 17 个领域的泰语法律时序问答基准
NitiBench+: A Temporal Thai Legal Question Answering Benchmark Across 17 Domains
AI 导读
研究者推出 NitiBench+，一个覆盖 17 个领域、含 1,670 道题的泰语法律问答基准，为每个被引法条同时保留判决当日有效文本与现行文本。对四家泰国法院 156,650 份判决的核查显示，最高 67% 引用了此后被修订或废止的法律；在 13 个开源权重模型上，读现行修订文本使 Bar-standard 分数下降 0.25，缺失已废止法律下降 0.79。
正文 · 原文
Abstract:Law changes over time. A court decides a case under the law in force on that day, and legal QA benchmarks grade answers against today's law. We checked 156,650 decisions of four Thai courts. Up to 67% of them cite law that has since been amended or repealed. We introduce NitiBench+, the sequel of our NitiBench, a Thai legal QA benchmark with 1,670 items across 17 domains, built from decisions of three courts. For each cited section it holds both the text in force on the decision date and the current text, and its retrieval corpus holds every dated version of every Thai statute section. Answers are graded on the Thai Bar Council rubric, the Bar-standard score, and on the overlap between the cited sections and the court's. The report has two parts. Part I measures the cost of the wrong text on 13 open-weight models. Reading the current text of amended law drops the Bar-standard score by 0.25 and missing repealed law drops it by 0.79. The penalty comes from rule changes, 0.45 on the 428 rule-changed items and none on the 331 wording-only items, and Citation F1 does not see it. The text in force is the fair setting. Part II benchmarks today's models on that setting. A public leaderboard ranks 26 models on a held-out test split of 500 items stratified by domain, with a validation split of the same shape released for public use, from memory alone, with the court's sections provided, and as a retrieval agent over the law in force, with the cost per item and the hallucinated citations. The best model scores 7.78 out of 10 and the best open-weight model 6.39. Without a search tool only five open-weight models recall any Thai law, and with it twenty of the 26 models gain and six lose. Leaderboard will take new models as they are released. Legal benchmarks must track dates, and on the right text today's models still leave a wide gap on Thai law.
| Comments: | Under review |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.03576 [cs.CL] |
| (or arXiv:2611.03576v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.03576 arXiv-issued DOI via DataCite |
Submission history
From: Pawitsapak Akarajaradwong [view email]
[v1]
Mon, 2 Nov 2026 18:28:39 UTC (1,808 KB)
[v2]
Tue, 3 Nov 2026 18:07:26 UTC (1,820 KB)
来源：arXiv cs.CL · arxiv.org