arXiv cs.AI· Yueer Pan, Yuexi Pan·· 13 小时前AI 评分31
思维链忠实性评估中的正确性条件选择问题
Correctness Conditioning as Selection in Chain-of-Thought Faithfulness Evaluation
AI 导读
将思维链（CoT）忠实性评估限定在回答正确的样本上会改变所测量的总体，研究用七变量结构因果模型推导出观测差距 Δ = Cov(A, C)/P(C=1)，区分难度混杂与捷径引发的对撞偏差。
正文 · 原文
Abstract:Restricting chain-of-thought (CoT) faithfulness evaluation to correctly answered instances changes the population being measured. We study this selection step in a seven-variable structural causal model and derive the exact observed gap $\Delta = \operatorname{Cov}(A, C)/P(C=1)$. The model distinguishes difficulty confounding from shortcut-induced collider bias and states the conditional-independence assumption needed to express the gap through latent response propensities. A protocol review of 18 papers distinguishes answer-correctness restrictions from agreement-based selection, benchmark construction, and training-data filtering. We evaluate four sub-ceiling deployments on arithmetic tasks and observe positive selection gaps in all four. The primary GLM-5.3 run yields a +7.4 percentage-point gap ($n = 1000$, $p < 0.0001$), and pooled Cochran-Mantel-Haenszel analysis gives OR = 1.70 ($p < 10^{-6}$). An assumption-free covariance decomposition assigns 42.8% of the GLM-5.3 gap to between-stratum variation in interventionable-operation count, leaving the majority unattributed. When counterfactual compliance is observable for every instance, evaluating all traces directly estimates population faithfulness without correctness filtering.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.37070 [cs.AI] |
| (or arXiv:2610.37070v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37070 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuexi Pan [view email]
[v1]
Tue, 27 Oct 2026 11:35:29 UTC (4,384 KB)
来源：arXiv cs.AI · arxiv.org