面向香港 AEC 行业的消防安全规范解读基准

Towards a Benchmark for Fire Safety Code Interpretation in the Hong Kong AEC Industry

AI 导读

研究团队与领域专家合作构建了香港消防安全规范的本地化基准数据集，包含选择题和开放式问题，用于评测 12 个主流闭源与开源模型。表现最好的闭源模型在选择题上达到 80-90% 准确率，开源模型普遍落后；但在开放式评测中，专家人工评估显示准确率仅为 50-70%，表明 LLM 在专业合规任务上仍有较大提升空间。

正文 · 原文

View PDF HTML (experimental)

Abstract:Large Language Models (LLMs) are increasingly explored in the Architecture, Engineering, and Construction (AEC) domain, yet their reliability in regulatory compliance remains uncertain. In this study, we evaluate the performance of LLMs on Hong Kong fire safety codes. In collaboration with domain experts, we create a localized benchmark dataset comprising both multiple-choice questions (MCQs) and open-ended questions (OEQs) to evaluate 12 widely used proprietary and open-source models. Our findings show that top-performing proprietary models achieve 80-90% accuracy on MCQs, whereas open-source models generally lag behind. However, on open-ended evaluation, expert human assessments reveal an accuracy range of only 50-70%, demonstrating that LLMs still have substantial room for improvement in specialized compliance tasks.

Subjects:	Computational Engineering, Finance, and Science (cs.CE); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2610.16127 [cs.CE]
	(or arXiv:2610.16127v1 [cs.CE] for this version)
	https://doi.org/10.48550/arXiv.2610.16127 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Le Qiu [view email]
[v1] Mon, 12 Oct 2026 03:45:26 UTC (237 KB)

来源：arXiv cs.AI · arxiv.org