arXiv cs.CY· Joan C. Timoneda, Sebasti\'an Vallejo Vera·· 11 小时前AI 评分31
政治与社会科学研究中使用大语言模型的可复制性、可重现性与稳健性
Replicability, Reproducibility, and Robustness in Political and Social Science Research with Large Language Models
AI 导读
针对 LLM 输出具有概率性、随模型和版本变化并影响下游统计结果的问题，该研究区分了可复制性、可重现性与稳健性，提出基于结果分布而非相同输出的概率性标准。研究引入标准化提示词脚本，测试编码器与解码器的模型内变异，并开发误差结构分析以诊断模型出错位置及误差是否一致。基于三个数据集，测量选择可逆转下游回归系数的符号；研究提出开放权重模型的堆叠集成，在保持估计有效性的同时提升校正步骤的精度。
正文 · 原文
Abstract:This chapter addresses the central reproducibility problem of research with large language models. LLM output is probabilistic and varies across models and versions, which can have significant effects on downstream statistical results. The chapter distinguishes between replicability, reproducibility, and robustness, and proposes a probabilistic standard based on the distribution of results, not identical outputs. It introduces a standardized prompt script, tests within-model variation for encoders and decoders, and develops error-structure analysis to diagnose where models err and whether their errors align. Using three datasets, the chapter shows that measurement choice can reverse the sign of downstream regression coefficients. Existing design-based corrections with gold-labeled subsamples restore validity, but come at the cost of estimate precision. The main contribution of the chapter is a stacked ensemble of open-weight models that improves precision in the correction step while retaining estimate validity.
| Comments: | Chapter 5 of "LLMs in Political and Social Science: Principles and Applications of Transformer Models," Oxford University Press, forthcoming 2027. Data and code: this https URL |
| Subjects: | Computation and Language (cs.CL); Computers and Society (cs.CY) |
| Cite as: | arXiv:2611.06639 [cs.CL] |
| (or arXiv:2611.06639v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.06639 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Joan C. Timoneda [view email]
[v1]
Wed, 4 Nov 2026 15:20:03 UTC (230 KB)
来源：arXiv cs.CY · arxiv.org