arXiv cs.AI· Hashmath Shaik, Gnaneswar Villuri, Alex Doboli·· 10 天前AI 评分24
能否预测 LLM 评审的位置翻转？基于残差流激活的线性探针研究
Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations
AI 导读
研究发现，LLM 评审在候选回答顺序调换后可能改变判定，而初始判定前的残差流激活可预测这种位置翻转。在 JudgeBench 的 534 对样本上，针对三个 Qwen3 评审和 Llama-3.1-8B 的线性探针 AUROC 达 .621-.850，比融合语言化置信度、判定标签 logits、回答长度和初始选择的基线高 .062-.113。
正文 · 原文
Abstract:The order in which candidate responses are presented can change an LLM judge's verdict. Detecting such a position flip ordinarily requires judging each pair in both orders, which doubles the number of judgments. We investigate whether residual stream activations recorded immediately before the initial verdict can predict a flip. We use nested grouped cross-validation to evaluate regularized linear probes on 534 JudgeBench pairs for three Qwen3 judges and Llama-3.1-8B. The linear probes achieve AUROCs of .621-.850 and outperform a combined baseline that uses verbalized confidence, verdict-label logits, response lengths, and the judge's initial choice by .062-.113 AUROC. Linear probes trained on JudgeBench and then frozen achieve AUROCs of .685-.853 on 1,802 MT-Bench comparisons without MT-Bench fitting or recalibration. These results show that pre-verdict activations support prediction of susceptibility to candidate order and outperform the non-activation predictors evaluated here.
| Comments: | It was submitted to neurips workshop (JUDGE workshop) and received a review score 6.5 combined with one strong accept and one above threshold accept |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07115 [cs.LG] |
| (or arXiv:2610.07115v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07115 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hashmath Shaik [view email]
[v1]
Mon, 5 Oct 2026 16:10:03 UTC (13 KB)
来源：arXiv cs.AI · arxiv.org