arXiv cs.CL· Daniel Robert Kling Alexander, Catherine Louise Kling·· 10 天前AI 评分27
无标准答案时如何评估语言模型：陈述偏好经济学的启示
Validity Without Ground Truth: What Stated-Preference Economics Offers the Evaluation of Language Models
AI 导读
研究者提出用陈述偏好经济学的效度框架评估大语言模型，涵盖内容效度、构念效度、效标效度、信度、激励兼容性与后果性等概念。在一项已发表的水质估值调查（Vossler et al. 2023）中测试六个模型，效度检验以经济理论预测形式呈现：需求曲线应向下倾斜，支付意愿应随商品范围和收入变化。
正文 · 原文
Abstract:Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should choose, how to weigh competing values. Stated-preference economics has faced this problem for decades. It judges survey responses without knowing the true value, through a framework of validity and related concepts: content, construct, and criterion validity, reliability, incentive compatibility, and consequentiality. We argue that this framework is a general method for evaluating language models, and we set out what each concept means for LLM evaluation. We demonstrate the approach using a published water-quality stated preference economic valuation survey (Vossler et al. 2023) administered to six models. In this economic application, the validity tests take the form of predictions from economic theory: demand should slope down, and willingness to pay should respond to the scope of the good and to income. The tests separate the models sharply. Two older models fail the most basic test at a household income level of \$75,000, and the two newest pass every test of theoretical validity we can score, but diverge on convergent validity. Passing validity tests shows that a model's answers are coherent, not that they are correct.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); General Economics (econ.GN) |
| Cite as: | arXiv:2610.10506 [cs.AI] |
| (or arXiv:2610.10506v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10506 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Daniel Alexander [view email]
[v1]
Wed, 7 Oct 2026 17:51:07 UTC (96 KB)
来源：arXiv cs.CL · arxiv.org