arXiv cs.CY· Elena Senger, Yuri Campbell, Rob van der Goot, Barbara Plank·· 17 小时前AI 评分42
高分下的隐藏失败：评估基于编码手册的 LLM 内容分析
High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis
AI 导读
一项覆盖四个数据集、五个模型的大规模实证分析发现，基于编码手册的 LLM 内容分析流程在标准指标下高分频现，却同时存在严重的标签级差异、多次运行不稳定、领域敏感和标签解释漂移等失败模式。这些失败模式在准确率和 micro-F1 等聚合指标下几乎不可见，且在常用优化策略下依然存在。研究结论是，当前 LLM 编码流程在缺乏针对性验证时不能被视为稳定的测量工具。
正文 · 原文
Abstract:Large language models (LLMs) are increasingly used for codebook-based content analysis in computational social science, where codebooks translate theoretical concepts into labeling schemes. While prior work reports strong predictive performance, it remains unclear whether these systems produce reliable and robust text annotations. We conduct a large-scale empirical failure-mode analysis of LLM-based coding pipelines across four datasets and five models by systematically evaluating their stability, robustness, and conceptual validity under controlled perturbations, domain shifts, and optimization procedures. We uncover multiple failure modes that are largely invisible under standard metrics, showing that high aggregate scores frequently coexist with severe label-level disparities, instability across runs, domain sensitivity, and shifts in label interpretation. Importantly, these failure modes persist under widely used optimization strategies and remain largely invisible under standard metrics such as accuracy and micro-F1. Our results demonstrate that current LLM-based coding pipelines cannot be treated as stable measurement tools without targeted validation beyond aggregate performance.
| Comments: | EMNLP Findings 2026 |
| Subjects: | Computation and Language (cs.CL); Computers and Society (cs.CY) |
| Cite as: | arXiv:2610.35179 [cs.CL] |
| (or arXiv:2610.35179v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35179 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Elena Senger [view email]
[v1]
Mon, 26 Oct 2026 10:40:54 UTC (170 KB)
来源：arXiv cs.CY · arxiv.org