arXiv cs.CL· Ethan Zhao, Maksym Taranukhin, Wei Cui, Moira Aikenhead, Vered Shwartz·· 13 小时前AI 评分26
CanLegalRAGBench：评估加拿大判例法上的检索增强生成
CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Law
AI 导读
研究者提出 CanLegalRAGBench，一个基于真实查询和专家标注答案、扎根判例法的加拿大法律问答基准，用于评估法律 RAG 系统。评测显示检索性能对设计选择敏感，开源嵌入模型与闭源模型表现相当，但自动评估会惩罚系统检索到其他相关文档。生成答案常偏离标准答案，8-29% 的陈述未获检索文档支持。
正文 · 原文
Abstract:RAG-based legal assistants have been growing in popularity, but LLM hallucinations remain a key issue and potentially undermines justice. While benchmarks have been developed to evaluate progress, many rely on synthetic queries rather than realistic legal scenarios. Moreover, Canadian law remains underrepresented in existing evaluations. To address this gap, we introduce CanLegalRAGBench, a Canadian legal QA benchmark based on realistic queries and expert-annotated answers grounded in case law. Our evaluation shows that retrieval performance is sensitive to design choices and that open-source embedding models are competitive with closed source models. However, it also reveals the limitation of automatic evaluations that penalize systems for retrieving alternative relevant documents. We also find that generated answers often diverge from gold responses, either with hallucinations or by producing overly detailed or irrelevant content, with 8-29% of claims not being supported by the retrieved documents. We hope this benchmark will help drive continued progress in addressing limitations of legal RAG systems.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.30497 [cs.CL] |
| (or arXiv:2605.30497v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.30497 arXiv-issued DOI via DataCite |
Submission history
From: Ethan Zhao [view email]
[v1]
Thu, 28 May 2026 19:24:23 UTC (2,761 KB)
[v2]
Tue, 18 Aug 2026 20:08:56 UTC (2,785 KB)
[v3]
Wed, 21 Oct 2026 17:36:42 UTC (2,786 KB)
来源：arXiv cs.CL · arxiv.org