arXiv cs.CL· Rabindra Lamsal, Sisi Zlatanova, Xuesong Shen·· 13 小时前AI 评分18
IFC-MCQ：为评估 LLM 智能体的 IFC 数据检索自动生成多选题基准
IFC-MCQ: Automatically Generating Multiple-Choice Benchmarks for Evaluating LLM Agents on IFC Data Retrieval
AI 导读
IFC-MCQ 可从 IFC 文件自动生成多选题基准，用精确的选项集评分替代需要人工或 LLM 审阅的自由文本评估。其分类体系覆盖 4 组、15 个类别和 36 个子类型，答案键取自文件本身，干扰项由文件派生值或预定义规则构造。在规模与建模特征各异的 IFC 模型上评测两种智能体配置后发现，性能会随源 IFC 模型和检索接口变化，并暴露出类别特有的失败模式。
正文 · 原文
Abstract:Industry Foundation Classes (IFC), the standard for building information exchange in architecture, engineering, and construction (AEC), is increasingly queried through large language models (LLMs) using natural language. Evaluating these systems at scale is difficult because free-text answers require costly human or LLM-based review. We propose IFC-MCQ, a method that automatically generates multiple-choice benchmarks from IFC files for exact option-set grading. The taxonomy covers 15 categories and 36 subtypes across four groups, with answer keys derived from the file and distractors constructed from file-derived values or predefined rules. We applied IFC-MCQ to IFC models of varying sizes and modelling characteristics to assess generation coverage and evaluated two agentic configurations on selected models using tool- and code-based retrieval. The evaluation exposed category-specific failure modes and showed that performance can vary with the source IFC model and retrieval interface. IFC-MCQ enables scalable benchmarking of LLM-based IFC retrieval in AEC workflows.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.30533 [cs.CL] |
| (or arXiv:2610.30533v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.30533 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Rabindra Lamsal [view email]
[v1]
Wed, 21 Oct 2026 23:38:01 UTC (5,773 KB)
来源：arXiv cs.CL · arxiv.org