跳到正文
arXiv cs.CL· Teng Lin, Nan Tang·· 9 小时前AI 评分31

MM-ReportBench:面向表格多模态报告生成的综合基准

MM-ReportBench: A Comprehensive Benchmark for Multimodal Report Generation from Tables

AI 导读

MM-ReportBench 是覆盖 185 张真实表格、386 个任务的多模态报告生成评测基准,每项任务配有人工校验的参考报告。其双评审框架显示,20 个 SOTA 模型中最高分仅 68.9,38.5% 的报告存在数值幻觉,真正新颖的洞察稀少。消融实验表明单评审打分比双评审高 8-12 分。

正文 · 原文

View PDF HTML (experimental)

Abstract:The ability to generate multimodal reports integrating fluent narratives with faithful visualizations from tabular data, has become a cornerstone capability for modern vision-language models in practical domains such as finance, healthcare, and policy making. However, existing benchmarks are limited by single-modality focus, simplistic metrics, novelty evaluation, and visualization quality. We address these limitations with MM-ReportBench, a multimodal report evaluation benchmark covering 386 tasks across 185 real-world tables. Each task is paired with an expert-curated reference report featuring standardized chapter structures and verifiable key insight points, constructed through a hybrid pipeline of multimodal reports generation based on Monte Carlo Tree Search and rigorous human quality control. For robust evaluation, we design a dual-judge evaluation framework: Text Judge Model assesses textual quality along structural completeness, grounded insight verification with numerical fact-checking against source tables, and novelty analysis that distinguishes genuine analytical discovery from trivial paraphrasing; Visualization Judge Model evaluates generated visualizations across data fidelity, expressiveness, aesthetics, and technical correctness. They also evaluate cross-modal consistency. Extensive experiments on 20 state-of-the-art models reveal critical weaknesses: even the strongest model achieves only 68.9 overall score, with numerical hallucination affecting 38.5% of reports and genuinely novel insights remaining rare. Our ablation studies further demonstrate that single-judge evaluation consistently inflates scores by 8-12 points compared to dual-judge assessment, confirming that rigorous and reliable benchmarking demands multi-faceted, complementary evaluation probes.
Subjects: Computation and Language (cs.CL)
Cite as: arXiv:2611.22514 [cs.CL]
  (or arXiv:2611.22514v1 [cs.CL] for this version)
  https://doi.org/10.48550/arXiv.2611.22514

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Teng Lin [view email]
[v1] Tue, 17 Nov 2026 10:06:56 UTC (3,713 KB)

来源:arXiv cs.CL · arxiv.org