arXiv cs.CL· Nils Schwager, Christoph Hau, Simon M\"unker, Achim Rettinger·· 12 小时前AI 评分31
The Unsampled Truth：量化语言模型心理测量中的提示词伪影
The Unsampled Truth: Quantifying Prompt Artifacts in LM Psychometrics
AI 导读
研究提出一种诊断设计，交叉五种语义不同的基线人格与四类提示词组件（人格措辞、任务指令、题目措辞、选项符号）的各五种语义等价变体，通过 1-Wasserstein 距离分离目标效应与提示词伪影。
正文 · 原文
Abstract:When prompting language models for psychometric assessment, researchers assume that the responses reflect the injected persona and the meaning of the survey item. We test this premise using a diagnostic design that crosses five semantically distinct baseline personas with five semantically equivalent variants of each of four prompt components (persona wording, task instruction, item wording, option symbol). Measuring the 1-Wasserstein distance between the resulting response distributions and partitioning the variation among the five components allows for the separation of target effects from prompt artifacts. We apply the framework to 13 open-weight small language models (0.6B to 14B) on the Big Five Inventory and the Short Dark Triad. We find that in most models, the task instruction and option symbol displace response distributions further than paraphrasing the persona description or the item itself. For a substantial share of items, the artifact share of explained variation exceeds 50%; non-semantic changes of the prompt account for more response variation than the baseline personas. Our framework lets researchers quantify these prompt artifacts before interpreting psychometric output.
| Comments: | 12 pages, 5 figures, 3 tables. Accepted to the Findings of EMNLP 2026, Budapest, Hungary |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.03357 [cs.CL] |
| (or arXiv:2606.03357v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.03357 arXiv-issued DOI via DataCite |
|
| Journal reference: | Findings of the Association for Computational Linguistics: EMNLP 2026 |
| Related DOI: | https://doi.org/10.18653/2026.findings-emnlp.1009
DOI(s) linking to related resources |
Submission history
From: Nils Schwager [view email]
[v1]
Tue, 2 Jun 2026 09:05:25 UTC (184 KB)
[v2]
Mon, 31 Aug 2026 13:56:58 UTC (166 KB)
[v3]
Thu, 5 Nov 2026 09:54:09 UTC (166 KB)
来源：arXiv cs.CL · arxiv.org