arXiv cs.CL· Xinyi Liu, Hooshang Nayyeri, Emine Yilmaz, Dilek Hakkani-Tur, Charith Peris, Hari Thadakamalla·· 3 天前AI 评分32
PersEvalBench：LLM 评估器能否判断人设信息的语境恰当使用？
PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?
AI 导读
研究者推出 PersEvalBench，用 403 个多轮场景测试 LLM 评估器能否识别用户历史信息的恰当与不当使用，覆盖欠个性化、过度个性化和状态误用三类共九种受控错误。在五个 LLM 评估器上，即便提供完整对话历史，错误识别率仍有限；人类对欠个性化错误的识别率为 94.4%，最强评估器仅 63.9%，且常把错误方向判反。高亮相关历史轮次仅带来小幅提升。
正文 · 原文
Abstract:Large language model (LLM) evaluators are increasingly used to assess personalized assistants, yet whether they can reliably judge personalization has rarely been tested. We introduce PersEvalBench, a benchmark testing whether an evaluator can recognize appropriate and inappropriate uses of user history. It contains 403 multi-episode scenarios where user information should persist, transfer, or cease to apply. The benchmark covers nine controlled error subtypes across three families: under-personalization, over-personalization, and misapplied-state errors. Human audits confirm that generated errors match their intended types and that the prescribed uses of user information are reasonable.
Across five LLM evaluators, recognition of controlled personalization errors remains limited even with the full dialogue history. On the human-evaluated subset, under-personalization shows the largest gap: humans recognize 94.4% of these errors, compared with 63.9% for the strongest reported evaluator. Evaluators often recognize that a response is problematic but reverse the error direction, mistaking failure to follow an applicable preference (UNDER) for use of an inapplicable preference (OVER). Easier evidence access offers little help: highlighting the relevant history turns raises error recognition only slightly, suggesting that the remaining difficulty lies in judging what role that information should play in the response.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19314 [cs.CL] |
| (or arXiv:2610.19314v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19314 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xinyi Liu [view email]
[v1]
Tue, 13 Oct 2026 23:44:58 UTC (922 KB)
来源：arXiv cs.CL · arxiv.org