arXiv cs.CL· Mingzhe Lu, Jinhao Liu, Yunpeng Li, Hang Su, Lichun Fan, Jian Luan·· 2 天前AI 评分18
SITAR：面向自动语音识别评估的情境失真度量
SITAR: A Situational Distortion Metric for Automatic Speech Recognition Evaluation
AI 导读
研究者提出 SITAR，一种面向自动语音识别（ASR）的情境失真度量，通过有界递归公式将情境严重度与累积范围聚合为统一分数，并训练为轻量级 BERT 家族回归器，单次前向传播即可为参考-假设对打分，无需运行时高昂成本。验证涵盖宏观与微观层面的外在效度及评分器内在可靠性，显示 SITAR 能跟踪模型能力扩展并提供正交诊断价值。完整代码、模型权重与数据集将在内部评审后发布。
正文 · 原文
Abstract:In automatic speech recognition (ASR), transcription errors with similar lexical or semantic discrepancies can have markedly different consequences for the situation conveyed by speech. Capturing these differences requires not merely contextual assessment, but an understanding of what the spoken discourse describes, establishes, or seeks to bring about. However, relying on LLM judges introduces computational cost and scoring variability. We introduce SITAR, a situational distortion metric for ASR, realized through a comprehensive end-to-end construction framework. Specifically, this framework aggregates situational severity and cumulative extent into a unified score via a bounded recursive formulation. Using targets derived from this formulation with limited offline LLM support, we train SITAR as a lightweight BERT-family regressor that scores reference-hypothesis pairs in a single forward pass, entirely free of any prohibitive runtime costs. Recognizing that introducing a new metric demands rigorous justification, we formalize a principled validation protocol spanning both extrinsic validity, assessed at macro and micro levels, and intrinsic scorer reliability. The former, evaluated across diverse datasets and models, demonstrates that SITAR reliably tracks broad capability scaling while providing orthogonal diagnostic utility, whereas the latter, examined on a purpose-built controlled benchmark, confirms the operational soundness of the scoring instrument itself. Through systematic validation, we demonstrate that SITAR holds strong potential as a practical, lightweight complement to existing ASR metrics. We will release the full recipes (code, model weights, dataset, etc.) after the internal review to support community research.
| Comments: | 33 pages, 4 figures, including appendices |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.21440 [cs.CL] |
| (or arXiv:2610.21440v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.21440 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Mingzhe Lu [view email]
[v1]
Thu, 15 Oct 2026 07:48:03 UTC (901 KB)
来源：arXiv cs.CL · arxiv.org