arXiv cs.AI· Yanki Margalit, Nurit Cohen-Inger, Erni Avram, Ran Taig, Seffi Cohen, Oded Margalit·· 12 小时前AI 评分34
Agents-as-Peers：将 LLM 评估变为多智能体同行评审的偏差控制协议
Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment
AI 导读
研究者提出 Agents-as-Peers 分布式评估协议，让每个 LLM 智能体同时生成任务、回答共享任务集并评审同伴答案，且不设特权评审者、排除自评。在 12 个 LLM、420 道智能体生成问题的实例中，每种评审机制聚合 55,440 条非自评同行评审。
正文 · 原文
Abstract:Large language model (LLM) evaluation commonly relies on fixed human-authored benchmarks or centralized human/model judges. Fixed benchmarks can age and enter model training corpora, while centralized judges introduce their own scoring preferences.
We cast LLM evaluation as a multi-agent peer-assessment problem and introduce Agents-as-Peers, a distributed evaluation protocol in which every agent generates tasks, answers the shared task set, and judges the answers of its peers, with no privileged evaluator.
Self-ratings are excluded from the ranking, and judging is repeated under controlled identity and presentation regimes so that self bias, identity effects, presentation-position sensitivity, and judge severity are measured rather than left implicit.
In an instantiation with 12 LLMs and 420 agent-generated questions, Agents-as-Peers aggregates 55,440 non-self peer judgments per regime.
Most agents assign their own answers higher scores than those answers receive from peer judges; visible identity changes scores for some agents; the first fixed answer slot is associated with scores 0.39 points higher than the shuffled baseline; and judges differ markedly in severity.
On labeled benchmarks, peer scores align strongly with accuracy on TruthfulQA (Pearson r=0.904) and GSM8K (r=0.873). On TruthfulQA, the peer-score correlation is substantially higher than the self-score correlation (r=0.538).
A benchmark-overlap screen and item-level human audit further characterize the endogenous task set.
These results support Agents-as-Peers as a cohort-relative measurement protocol whose ranking is reported together with the biases and rater effects that shape it.
Code and dataset: this https URL
| Comments: | Version 2.0 of PeerRank |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2602.02589 [cs.AI] |
| (or arXiv:2602.02589v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2602.02589 arXiv-issued DOI via DataCite |
Submission history
From: Nurit Cohen Inger [view email]
[v1]
Sun, 1 Feb 2026 06:01:28 UTC (1,765 KB)
[v2]
Tue, 20 Oct 2026 07:13:23 UTC (824 KB)
来源：arXiv cs.AI · arxiv.org