跳到正文
arXiv cs.AI· Julian Skirzynski, Harry Cheon, Shreyas Kadekodi, Meredith Stewart, Berk Ustun·· 10 小时前AI 评分24

为概念瓶颈模型构建合成基准,衡量真正重要的能力

Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models

AI 导读

研究者提出一套面向概念瓶颈模型的合成基准,可生成带标签数据集并控制数据模态、概念选择与标注质量等影响性能的属性,从而隔离模型失败原因。该基准已用于评估决策支持与自动化两大用例,能识别新的失败模式并归因于架构或数据。基准代码已公开,论文为 NeurIPS 2026 Evaluations and Datasets Track 相机稿。

正文 · 原文

View PDF HTML (experimental)

Abstract:Concept bottleneck models predict using high-level concepts detected in inputs. Although these models provide a simple way to reap benefits from interpretability, they are hard to train and evaluate because very few real-world datasets include concept labels. In this paper, we develop synthetic benchmarks for concept bottleneck models. With our benchmarks, researchers can generate labeled datasets and control properties that affect performance, such as data modality, the choice of concepts, or annotation quality. They can thus find problems that suit these models and isolate the factors behind their failures. We demonstrate how our benchmarks can be used to evaluate major use cases for concept bottleneck models: decision support, i.e., helping human experts make better decisions; and automation, i.e., handling routine tasks without supervision. Our demonstrations show how the benchmarks can identify new failure modes and attribute them to either the architecture or the data.
Comments: Camera-ready version, NeurIPS 2026 (Evaluations and Datasets Track). Benchmarks available at this https URL
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
ACM classes: I.2.6
Cite as: arXiv:2606.04326 [cs.LG]
  (or arXiv:2606.04326v2 [cs.LG] for this version)
  https://doi.org/10.48550/arXiv.2606.04326

arXiv-issued DOI via DataCite

Submission history

From: Julian Skirzynski [view email]
[v1] Wed, 3 Jun 2026 01:01:05 UTC (1,311 KB)
[v2] Tue, 17 Nov 2026 19:59:15 UTC (370 KB)

来源:arXiv cs.AI · arxiv.org