arXiv cs.AI· Shuyue Stella Li, Bhargavi Paranjape, Kerem Oktar, Zhongyao Ma, Gelin Zhou, Lin Guan, Na Zhang, Sem Park, Lin Chen, Diyi Yang, Yulia Tsvetkov, Asli Celikyilmaz·· 10 小时前AI 评分52
HorizonBench 发布:评测模型能否跟上用户偏好的长期变化
HorizonBench: Long-Horizon Personalization with Evolving Preferences
AI 导读
研究者提出 HorizonBench 及其数据生成器,模拟用户偏好随生活事件变化并生成六个月的对话,用于评测模型能否更新对用户的认知。基准包含 360 名用户的 4,245 个问题,偏好变化后旧偏好会作为干扰选项出现;评测的 25 个前沿模型都更倾向于选择过时偏好,最佳准确率为 52.8%。
正文 · 原文
Authors:Shuyue Stella Li, Bhargavi Paranjape, Kerem Oktar, Zhongyao Ma, Gelin Zhou, Lin Guan, Na Zhang, Sem Park, Lin Chen, Diyi Yang, Yulia Tsvetkov, Asli Celikyilmaz
Abstract:Personal assistants now talk with the same user for months, and users' preferences change as their lives change; an assistant that keeps serving a preference the user has outgrown fails that user. Real conversations do not record when or why a preference changed, and existing benchmarks either hold preferences fixed or have the user state the change, so they cannot tell whether a model forgot what the user said or failed to update its belief about the user. We introduce HorizonBench and its data generator, which samples how a user's preferences change and generates six months of conversation from that sequence; each change follows a life event that the user mentions without stating the new preference. A human longitudinal preference dataset that we collect and release, 83,024 ratings from 160 people over 13 weeks, sets how often and how much life events change preferences. HorizonBench contains 4,245 questions from 360 users, and when a preference has changed, its earlier value is one of the answer options. All 25 frontier models we evaluate choose the outdated preference more often than evenly spread errors would predict, and the best reaches 52.8% accuracy. Two of the newest models score no higher when given only the relevant conversations than with the full history, yet one added sentence stating the change raises their accuracy by 11 to 22 points, and retrieval and memory methods do not recover the lost accuracy. Remembering what a user said and knowing what the user prefers now are therefore separate capabilities, and HorizonBench, its generator, and the human dataset support research on both.
| Comments: | 26 pages, 6 figures, 19 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2604.17283 [cs.CL] |
| (or arXiv:2604.17283v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2604.17283 arXiv-issued DOI via DataCite |
Submission history
From: Shuyue Stella Li [view email]
[v1]
Sun, 19 Apr 2026 06:55:10 UTC (804 KB)
[v2]
Tue, 17 Nov 2026 22:59:02 UTC (735 KB)
来源:arXiv cs.AI · arxiv.org