arXiv cs.AI· Adi Krishnamoorthy, X. Y. Han·· 4 小时前AI 评分58
研究:推理投入提升 LLM 购物智能体的置信度而非正确率
Thinking Harder Doesn't Lead to Better Choices: Reasoning Effort Increases the Confidence but Not the Correctness of LLM Shopping Agents
AI 导读
研究在 WebShop、Amazon SQID 和 Wayfair WANDS 三个商品选择基准上测试十个前沿模型,发现提高推理投入并未在统计意义上提升决策质量,却显著提高了模型自报的置信度。研究还测量了各投入档位下的校准情况,发现多数模型校准不佳,且错误高度共享:一个模型在某任务出错时,其他多个模型往往同样出错,其中不少还选中同一个错误商品。
正文 · 原文
Abstract:Large language models (LLMs) are increasingly deployed as autonomous shopping agents that process multimodal product listings (consisting of visual and text data) and recommend purchases on behalf of consumers. Naturally, one might assume that models with more thinking power would make higher-quality purchasing decisions. To test this belief, we ask two questions about such agents: does asking a model to think harder make it choose better products, and does the confidence it reports tell us the likelihood it may be wrong? We benchmark ten frontier models on three product-choice benchmarks built from WebShop, Amazon's SQID and Wayfair's WANDS platforms to see if LLMs with more thinking power actually make better-quality product purchase decisions. Across all three benchmarks, we found that increasing the thinking effort does not increase decision quality in a statistically meaningful way. The same change in thinking effort raises stated confidence at a statistically significant level, so extra thinking buys confidence rather than correctness. We also measure the calibration of these models (how well their reported confidence percentage matches their true correctness percentage) in these tasks and show that most models are not well-calibrated at every effort level. We also find that errors are strongly shared: when one model errs in a task, several others tend to err on that same task. Furthermore, on tasks where several models all err, a large number of them pick the same wrong product.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY); Human-Computer Interaction (cs.HC); General Economics (econ.GN) |
| Cite as: | arXiv:2611.22033 [cs.AI] |
| (or arXiv:2611.22033v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22033 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Aditya Krishnamoorthy [view email]
[v1]
Tue, 17 Nov 2026 00:55:58 UTC (2,502 KB)
来源:arXiv cs.AI · arxiv.org