arXiv cs.CY· Oleg Vasilyev·· 11 小时前AI 评分34
聊天机器人 LLM 对总统行为与假设性改革的判断稳健性研究
Robustness of Chatbot LLM Judgments of Presidential Actions and Hypothetical Reforms
AI 导读
研究者在 30 天内通过消费级聊天界面，让 Mistral、Gemini Flash、Claude Sonnet 和 Claude Opus（两种 effort 设置）共五种配置每天两次为 40 项近四届美国总统行政行为和 15 项假设性改革打分。
正文 · 原文
Abstract:Chatbots are increasingly consulted about public affairs, and their judgments of policies are useful only if they are robust. We measure this robustness in the setting of an ordinary user, through consumer chat interfaces. For 30 days, five chatbot configurations (Mistral, Gemini Flash, Claude Sonnet, and Claude Opus at two effort settings) were asked twice a day to score the long-term effect on the United States of 40 actions of the last four presidential administrations and of 15 hypothetical reforms, each with a confidence and a short comment. The judgments are robust: all configurations agree on the sign and the ranking of the administrations' mean scores, and two different configurations asked on the same day give opposite signs to the same action in about 4% of cases. Responses from different days differ more than same-day responses (which shared the order of the items); the difference grows slowly over the month, except where the service itself changed. We identify three such changes, which shifted the judgments while leaving little or no trace in the interface. Treating the comments as a retrieval task, we find them specific to each item and consistent across runs and chatbots, with lexical and embedding similarities agreeing. We release the dataset of 16,500 judgments.
| Subjects: | Computation and Language (cs.CL); Computers and Society (cs.CY) |
| Cite as: | arXiv:2610.39398 [cs.CL] |
| (or arXiv:2610.39398v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.39398 arXiv-issued DOI via DataCite |
Submission history
From: Oleg Vasilyev [view email]
[v1]
Thu, 29 Oct 2026 05:32:29 UTC (538 KB)
[v2]
Fri, 30 Oct 2026 17:23:08 UTC (539 KB)
来源：arXiv cs.CY · arxiv.org