arXiv cs.AI· HyunJoon Jung, William Na·· 3 天前AI 评分29
LLM 智能体评审的分数与覆盖度为何脱钩：一项基于 960 场会话的研究
Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluation
AI 导读
研究通过 15 项任务、两组模型配对共 960 场会话发现，基于人格设定的 LLM 智能体评审在 Turing 式验证中与人类评分者难以区分。评审质量分数随评审团规模呈对数增长，而新问题发现量呈次线性幂律增长，分数饱和速度约为发现速度的两倍。这一脱钩源于评审团多样性：Big Five 人格条件让智能体探查不同质量维度，消融实验证实结构化人格设定而非简单提示词才是产生该缩放特性的必要条件。
正文 · 原文
Abstract:LLM-based agent judges are an emerging approach to evaluating conversational AI, yet a fundamental uncertainty remains: can we trust their assessments, and if so, how many are needed? Through 960 sessions with two model pairs across 15 tasks, we show that persona-based agent judges produce evaluations indistinguishable from human raters in a Turing-style validation. We then identify a score-coverage dissociation: quality scores improve logarithmically with panel size, while unique issue discoveries follow a sublinear power law-both exhibit diminishing returns, but scores saturate roughly twice as fast as discoveries. We hypothesize this reflects a power law distribution of the finding space: critical issues are discovered first by small panels, while corner cases require progressively larger panels, analogous to species accumulation curves in ecology. The mechanism traces to ensemble diversity-Big Five personality conditioning makes agents probe different quality dimensions, with expert judges acting as adversarial probes that push discovery into the tail of the finding distribution. A controlled ablation confirms that structured persona conditioning, not simple prompting, is required to produce these scaling properties.
| Comments: | Accepted to COLM 2026 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Human-Computer Interaction (cs.HC); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2604.00477 [cs.AI] |
| (or arXiv:2604.00477v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2604.00477 arXiv-issued DOI via DataCite |
Submission history
From: HyunJoon Jung [view email]
[v1]
Wed, 1 Apr 2026 04:44:21 UTC (22,089 KB)
[v2]
Sat, 10 Oct 2026 03:38:36 UTC (3,020 KB)
来源：arXiv cs.AI · arxiv.org