arXiv cs.CL· Tobias Hallmen, Elisabeth Andr\'e·· 11 小时前AI 评分31
基于评分工具锚定的 LLM 评委可跨三个德语语料迁移专家咨询质量评分
Instrument-Anchored LLM Judges Transfer Expert Counseling-Quality Ratings Across Three German Corpora
AI 导读
研究用四个小型开源 LLM 评委按专家评分工具中的构念打分，在三个德语模拟咨询语料（共 195 场会话）上实现跨域预测，留出语料 Spearman ρ 达 .533，比同域训练高出约五分之一（.420，配对差距 +.113）。信号主要来自语言内容：四个评委单独迁移为 .523，非言语模块仅增加 +.01。让评委输出评分工具构念而非通用对话质量，可将其迁移性提升约 30%（.32 到 .42）。
正文 · 原文
Abstract:Automatic assessment of communication quality in counseling conversations is bottlenecked by data: expert-rated corpora are small and expensive to grow. Therefore we study cross-domain prediction of the expert's overall impression across three German simulated-counseling corpora from one university (two general-practice medical, one parent-teacher; $n=195$ sessions). Features are construct scores that four small LLM judges assign from the transcript, drawn from the experts' own rating instruments, plus a compact nonverbal block; all models are open-weight and run locally, preserving data protection. Trained on the other two corpora, this stack predicts the held-out corpus at Spearman $\rho = .533$; in-domain training on the same sessions falls about a fifth short ($.420$; paired gap $+.113$, 95% CI $[+.067, +.157]$), and the gap holds at matched training size and on identical test sessions. The signal lies in what is said: the four judges alone transfer at $.523$, and the nonverbal block adds $+.01$ (CI includes zero). Asking a judge for instrument constructs instead of generic dialogue qualities raises its transfer by about 30% ($.32$ to $.42$); restoring speaker attribution where per-speaker audio was lost is worth $+.08$ to $+.10$ on that fold. Where a program's own ratings are scarce, other programs' data is a viable training set and language carries it.
| Comments: | Preprint. 22 pages, 3 figures |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.08055 [cs.CL] |
| (or arXiv:2610.08055v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.08055 arXiv-issued DOI via DataCite |
Submission history
From: Tobias Hallmen [view email]
[v1]
Tue, 6 Oct 2026 09:53:25 UTC (85 KB)
[v2]
Fri, 30 Oct 2026 16:11:30 UTC (91 KB)
来源：arXiv cs.CL · arxiv.org