arXiv cs.CL· Patrik Wolf, Thomas Kleine Buening, Andreas Krause, Celestine Mendler-D\"unner·· 10 小时前AI 评分34
语言模型中的划分、提示与聚合:统计自一致性研究
Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models
AI 导读
研究者提出用统计自一致性检验 LLM 的条件推断能力:以二叉树递归划分人群、用提示词描述子群体并聚合估计值,发现前沿模型普遍违反全概率定律等基本一致性性质。研究还发现"宏观谬误"——由更细粒度子群体回答重建的估计往往比直接的人群级估计更贴近人类参考数据,说明模型具备相关子群体知识却未能可靠地传播到聚合估计中。该工作已被 NeurIPS 2026 主会接收。
正文 · 原文
Abstract:Large language models (LLMs) are often used to perform a form of conditional inference. In-context learning is one implicit way to do this. The prompt specifies a context and the model's output is treated as an estimate of the corresponding conditional distribution. If this interpretation holds, then LLM estimates should satisfy basic probabilistic identities. In particular, the law of total probability asserts that prior-weighted conditional distributions aggregate into population-level marginals over any valid partition of the population. In this work, we investigate to what extent LLM estimates adhere to this self-consistency principle. We use binary trees as an evaluation scaffold to recursively partition a population into increasingly fine-grained subpopulations. We then prompt LLMs with verbalized subpopulation descriptions in context, aggregate the resulting estimates back into population-level estimates, and compare them across partitions of varying granularity. Translating these comparisons into a benchmark score and evaluating frontier models across problem domains, we demonstrate widespread violations of basic consistency properties. An in-depth study of persona prompting further reveals a pattern we call the macro fallacy: estimates reconstructed from more fine-grained subpopulation responses are often better aligned with human reference data than direct population-level estimates. The effect persists across variations in partitioning, verbalization, and estimation task, suggesting that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates. This gap establishes statistical self-consistency as an unsaturated, reference-free criterion for evaluating LLMs.
| Comments: | Accepted to NeurIPS 2026 main conference |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.15277 [cs.CL] |
| (or arXiv:2607.15277v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.15277 arXiv-issued DOI via DataCite |
Submission history
From: Patrik Wolf [view email]
[v1]
Thu, 16 Jul 2026 17:59:31 UTC (710 KB)
[v2]
Thu, 19 Nov 2026 18:38:26 UTC (230 KB)
来源:arXiv cs.CL · arxiv.org