arXiv cs.CL· Jewon Yeom, Jaewon Sok, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim·· 12 小时前AI 评分31
多语言 P(True) 自评估中问题、回答与验证器语言如何相互影响
Disentangling Question, Response, and Verifier Languages in Multilingual P(True) Self-Evaluation
AI 导读
研究在六语言因子设计中独立变化问题语言、回答语言与验证器指令语言，用 10 个 3B-14B 开源权重模型在 LSQA、MMMLU 和 TriviaQA 上以首 token P(True) 探针测量置信度。
正文 · 原文
Abstract:When a language model is asked whether its own answer is correct, three languages are involved: that of the question, that requested for the answer, and that of the verifier instruction asking for the judgment. Multilingual confidence evaluations seldom separate the three. We vary them independently in a six-language factorial design with ten open-weight models (3B-14B) on LSQA, MMMLU and TriviaQA, holding each answer fixed across verifier languages and measuring confidence with a first-token P(True) probe. The three languages move different parts of the evaluation: which of them governs accuracy depends on the benchmark, matching the response language to the question or verifier language sharpens discrimination but not calibration, and the verifier language alone shifts calibration error by up to 0.04 without changing accuracy. The scorer matters as well: an unvalidated judge would have shrunk the MMMLU English-Hindi accuracy gap from 0.198 to 0.038. Multilingual confidence results, model rankings included, are therefore comparable only when all three languages are reported and the scorer is validated.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.32144 [cs.CL] |
| (or arXiv:2610.32144v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32144 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Je Won Yeom [view email]
[v1]
Fri, 23 Oct 2026 04:20:43 UTC (119 KB)
来源：arXiv cs.CL · arxiv.org