arXiv cs.AI· Zhenyu Zhao, Nanshan Jia, Jihyeon Je, Yifu Tang, Alvin Chan, Michael Spedden, Michael V. Palleschi, Sui Huang, Jingshen Wang, Zeyu Zheng·· 10 天前AI 评分22
3D-DefectBench：面向细粒度 3D 生成缺陷的视觉语言模型评估流水线受控析因研究
3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects
AI 导读
研究者提出 3D-DefectBench，一个用于严格分析评估流水线的大规模基准，覆盖几何、纹理与提示词遵循三类共九种细粒度二元缺陷，并可选人类严重度标注。该基准通过平衡析因设计在 84 种推理设计中变化 VLM、相机协议、视觉输入与提示词模式，发现模型选择是与人类标注一致性的主导变异来源，而紧凑的六视图 RGB 协议与更密集视图及加入深度或法线通道的配置表现相当。
正文 · 原文
Abstract:Automated evaluation is essential for scaling generative 3D systems, where exhaustive human review is costly and slow. Yet the reliability of an automated judge depends on the full evaluation pipeline, including the vision-language model (VLM), asset rendering, visual evidence, task specification, and human reference labels. We introduce 3D-DefectBench, a large-scale benchmark for rigorous evaluation-pipeline analysis. It complements holistic ratings and pairwise preferences with nine fine-grained binary defects spanning geometry, texture, and prompt adherence, with optional human severity annotations. Using a balanced factorial design, we vary the VLM, camera protocol, visual input, and prompt schema across 84 inference designs, and validate the resulting conclusions on a broader set of frontier models. Model choice is the dominant source of variation in agreement with human labels, while other pipeline factors also influence agreement, interact with the model, and can alter the best configuration. A compact six-view RGB protocol performs comparably to denser view sets and configurations augmented with depth or normal channels, making it a strong cost-effective default. Under this fixed design, the best of 12 VLMs still trail trained human labelers, and texture agreement drops sharply from expert-agreement to noisier silver labels. Severity annotations further show that binary judges recover most defects humans flag as severe. These results highlight the importance of evaluating automated judges as complete pipelines and calibrating them across human reference regimes.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Graphics (cs.GR) |
| Cite as: | arXiv:2607.10826 [cs.CV] |
| (or arXiv:2607.10826v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2607.10826 arXiv-issued DOI via DataCite |
Submission history
From: Zhenyu Zhao [view email]
[v1]
Sun, 12 Jul 2026 16:41:24 UTC (11,928 KB)
[v2]
Tue, 6 Oct 2026 05:25:52 UTC (14,150 KB)
来源：arXiv cs.AI · arxiv.org