arXiv cs.CL· Shuhaib Mehri, Philippe Laban, Sumuk Shashidhar, Marwa Abdulhai, Sergey Levine, Michel Galley, Dilek Hakkani-T\"ur·· 11 小时前AI 评分31
如何测量并缩小真实用户与模拟用户行为之间的分布差距
Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors
AI 导读
研究者提出一种测量真实与模拟用户行为分布差距的方法：从对话中提取用户行为表征，经聚类量化为离散分布后计算散度指标。该方法首次系统评估了 24 个基于 LLM 的用户模拟器在编程与写作任务上的表现，发现其与真实用户存在较大分布差距，且差距随模型族、规模和行文维度而变化。将行为互补的模拟器组合使用，可使整体分布比单一模拟器更接近真实用户。
正文 · 原文
Abstract:As user simulators are increasingly used for interactive training and evaluation of AI assistants, it is essential that they represent the diverse behaviors of real users. While existing works train user simulators to generate human-like responses, whether they capture the broad and heterogeneous distribution of real user behaviors remains an open question. In this work, we introduce a method to measure the distributional gap between real and simulated user behaviors, validated through a human study and ablations. Given a dataset of real and simulated conversations, our method extracts representations of user behavior from each conversation, quantizes them into discrete distributions via clustering, then computes divergence metrics. We provide the first systematic evaluation of 24 LLM-based user simulators on coding and writing tasks, and reveal a large distributional gap from real users that varies across model families, scales, and behavioral facets. Pairwise comparisons show that most simulators behave similarly, while a few stand apart. Combining behaviorally complementary simulators brings the resulting distribution closer to real users compared to either simulator on its own. Finally, a TF-IDF analysis of the clusters surfaces interpretable patterns of behaviors that simulators capture, miss, and hallucinate.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.07847 [cs.CL] |
| (or arXiv:2605.07847v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07847 arXiv-issued DOI via DataCite |
Submission history
From: Shuhaib Mehri [view email]
[v1]
Fri, 8 May 2026 15:09:25 UTC (15,460 KB)
[v2]
Fri, 6 Nov 2026 08:52:32 UTC (15,460 KB)
来源：arXiv cs.CL · arxiv.org