相同语义、不同判断：人类偏好标注如何随文本与音频模态变化

View PDF HTML (experimental)

Abstract:Preference-based reinforcement learning (PbRL) is the dominant framework for aligning AI systems to human preferences. However, evaluation protocols for such data were designed for text and have not been validated for speech. We present the first intraclass correlation coefficient (ICC)-based, controlled cross-modal study of human and synthetic preference annotations, comparing text and audio evaluations of identical semantic content across 100 prompts. We show that achieving \textit{good} (audio) to \textit{near good} (text) agreement within either modality (ICC(2,$k$) $\approx$ .81 audio, .74 text) requires $\sim$9 raters and modalities show marked differences in how people report preferences: audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria, with near-chance cross-modality agreement. We demonstrate that synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations. Together, these findings argue that evaluation protocols for audio preference data require modality-specific design rather than direct adaptation from text. Code, data, and the audio corpus are available at: this https URL

Comments:	Accepted to NeurIPS 2026 Evaluations and Datasets Track
Subjects:	Sound (cs.SD); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)
Cite as:	arXiv:2602.22710 [cs.SD]
	(or arXiv:2602.22710v4 [cs.SD] for this version)
	https://doi.org/10.48550/arXiv.2602.22710 arXiv-issued DOI via DataCite

Submission history

From: Aaron Broukhim [view email]
[v1] Thu, 26 Feb 2026 07:34:15 UTC (148 KB)
[v2] Wed, 6 May 2026 21:24:15 UTC (260 KB)
[v3] Tue, 3 Nov 2026 17:54:51 UTC (275 KB)
[v4] Wed, 4 Nov 2026 18:43:22 UTC (275 KB)