arXiv cs.AI· Grant Nasir Brown, Abdul Basir Bari, Anmol Tadikonda, Isabel Song·· 1 天前AI 评分38
P-ContextBench：从个人事实回忆到个性化智能体行为
P-ContextBench: From Personal Fact Recall to Personalized Agent Behavior
AI 导读
研究者提出 P-ContextBench，一个基于人口结构种子构建、以人生故事为根基的合成数字人基准，用于评估 LLM 对个人上下文的理解。该基准包含 100 个角色、10,088 条隐藏事实、18,261 条应用痕迹片段和 4,810 个已验证测试用例，覆盖单源/多源事实恢复、时序状态追踪、高层个人模式推断及约束冲突下的安全处理五项能力。
正文 · 原文
Abstract:Large language models (LLMs) are increasingly used for personal tasks, yet evaluating their safe use of personal context remains difficult because such contexts are private, fragmented, dynamic, and socially situated. We introduce P-ContextBench, a realistic, verifiable, and scalable benchmark for evaluating personal-context understanding. P-ContextBench constructs population-structured, life-story-grounded digital humans from demographic seeds reflecting the U.S. population. Rather than using explicit, static profiles, P-ContextBench embeds traits, routines, goals, and constraints as implicit evidence across fragmented, evolving synthetic mobile-app records. It evaluates five capabilities: single-source fact recovery, multi-source fact recovery, temporal state tracking, inference of higher-level personal patterns from routine behavior, and safe handling of requests conflicting with persona-specific constraints through compliance, warning, clarification, or refusal. A multi-stage validation pipeline checks persona plausibility, context recoverability, information leakage, evidence preservation, test-case grounding, and human-judge agreement. We instantiate P-ContextBench with 100 personas, 10,088 hidden facts, 18,261 app-trace fragments, and 4,810 verified test cases. The pipeline supports configurable generation across scales and demographic groups. Cost-effective state-of-the-art (SOTA) LLMs relevant to personal agents achieve below 60% accuracy on temporal tracking, higher-level pattern inference, and constraint-aware action-boundary decisions, with fewer than 10% of test cases answered correctly across all three repeated runs. Even a high-performing SOTA model remains inconsistent on these capabilities. P-ContextBench provides measurable targets for improving the accuracy, reliability, and safety of LLM-based personal agents.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.40959 [cs.AI] |
| (or arXiv:2610.40959v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.40959 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Isabel Song Dr. [view email]
[v1]
Thu, 29 Oct 2026 20:38:32 UTC (165 KB)
来源：arXiv cs.AI · arxiv.org