AU-Harness：面向 Audio-LLM 高效统一评估的开源工具包
AU-Harness: An Open-Source Toolkit for Efficient and Unified Evaluation of AUdio-LLMs
AU-Harness 是面向大型音频语言模型（LALM）的开源评估框架，评估处理时间较现有工具包减少 77%。它提供标准化评估协议与灵活配置，支持多轮对话动态分析，弥补现有框架在多轮上下文整合与规模化评估上的不足。该工具已被 EMNLP 2026 Findings 收录。
Authors:Hoang Nguyen, Sidharth Surapaneni, Akshay Kalkunte, Jash Mehta, Aman Tiwari, Oluwanifemi Bamgbose, Khyati Mahajan, Jash Shah, Shruthan Radhakrishna, Sathwik Tejaswi Madhusudhan, Vikas Yadav, Sai Rajeswar
Abstract:Large Audio Language Models (LALMs) are rapidly advancing, but evaluating them remains challenging due to inefficient and nonstandardized toolkits that limit fair comparison and systematic assessment. Existing evaluation frameworks exhibit three critical limitations: (1) slow and inefficient processing pipeline that bottlenecks large-scale studies, (2) inadequate multi-turn dialogue support, leaving fundamental questions about cross-turn context integration and performance dynamics over extended conversations in LALMs unanswered; and (3) the absence of unified and scalable evaluation framework capable of keeping pace with the rapid growth of both LALMs and audio benchmarks. To address these issues, we introduce AU-Harness, an efficient and comprehensive evaluation framework for LALMs. Our empirical studies demonstrate that AU-Harness achieves 77% reduction in evaluation processing time over existing evaluation toolkits, enabling large-scale LALM evaluations previously considered impractical. We provide standardized evaluation protocols and flexible configurations for fair model comparison across diverse scenarios. AU-Harness unlocks a range of in-depth analyses difficult to conduct without a unified foundation, including multi-turn dialogue dynamics, enabling the study of true audio reasoning capabilities in existing LALMs. AU-Harness provides both practical evaluation tools and insights into model limitations, advancing systematic LALM development.
| Comments: | EMNLP 2026 Findings |
| Subjects: | Sound (cs.SD); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2509.08031 [cs.SD] |
| (or arXiv:2509.08031v4 [cs.SD] for this version) | |
| https://doi.org/10.48550/arXiv.2509.08031 arXiv-issued DOI via DataCite |
Submission history
From: Aman Tiwari [view email]
[v1]
Tue, 9 Sep 2025 15:30:40 UTC (636 KB)
[v2]
Thu, 11 Sep 2025 16:27:59 UTC (634 KB)
[v3]
Mon, 11 May 2026 14:29:23 UTC (5,367 KB)
[v4]
Fri, 23 Oct 2026 04:45:20 UTC (6,188 KB)
来源：arXiv cs.AI · arxiv.org