arXiv cs.CL· Yuqin Dai, Zhanwei Zhang, Honghui Yang, Zhouheng Yao, Yueqi Zhang, Jingyu He, Chaoyang Shi, Fanding Wei, Boyuan Pan, Kai Xu, Yao Hu, Yongliang Shen·· 11 小时前AI 评分38
CARTBench：面向复杂模糊购物需求的评分标准评测基准
CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines
AI 导读
研究团队推出 CARTBench，用于评测 LLM 购物助手处理带时间线的复杂模糊需求（CART）的能力，覆盖 17 个零售领域、1,603 条商品记录与 1,328 条专家标注评分标准。
正文 · 原文
Authors:Yuqin Dai, Zhanwei Zhang, Honghui Yang, Zhouheng Yao, Yueqi Zhang, Jingyu He, Chaoyang Shi, Fanding Wei, Boyuan Pan, Kai Xu, Yao Hu, Yongliang Shen
Abstract:Large language model (LLM) shopping assistants now recommend products in multi-turn conversations. However, reliable evaluation faces Data scarcity and Difficulty Standardizing Criteria. To address these obstacles, we introduce CARTBench for CART (Complex and Ambiguous Requests with Timelines). For Data scarcity, we curate cases from real commercial shopping needs across 17 retail domains, pairing timestamped dialogue with 1,603 product records. For Difficulty Standardizing Criteria, we develop case-specific rubrics guided by Ideal Response Principles, distilled from excellent replies manually selected from successfully completed orders. These principles also guide Principle-Guided CART Prompting, our structured recommendation strategy. The experimental benchmark contains 1,328 expert-annotated criteria across Core, Excellence, and Safety. To answer the CART question, we evaluate 7 flagship commercial API models, including GPT-6.1-Sol and Claude-Opus-5, and 35B+ parameters open-weight models. In the first full-suite evaluation under the frozen rubrics, 3 of 10 models exceed 90% on Core, while none exceeds 45% on Excellence. Therefore, we identify an understanding-to-execution gap between required behavior and high-quality purchase advice. To examine these limitations, we diagnose shared failures in product evidence and unresolved needs, linked to changing requirements and decisive unknowns. Code and data are available at this https URL.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.05862 [cs.CL] |
| (or arXiv:2611.05862v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05862 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Dai Yuqin [view email]
[v1]
Wed, 4 Nov 2026 02:52:07 UTC (4,721 KB)
来源：arXiv cs.CL · arxiv.org