arXiv cs.AI· Cheng Tang, Kun Wang, Yaxuan Yang, Shiyu Wang, Allan Cohen, Rachael Eriksen Brown, Chandra Hawley Orrill·· 12 小时前AI 评分24
Smart Coder 框架：用 LLM 做质性编码的人机协作三阶段方法
Scaling Qualitative Analysis with Large Language Models: Development and Validation of the Smart Coder Framework for Human-LLM Collaborative Coding
AI 导读
研究者提出 Smart Coder 框架，通过 Instruction、Prompt Strategy、Reliability Check 三阶段结构化人机协作质性编码，并在 72 份中学数学教师访谈转录文本上验证。
正文
Abstract:Current uses of large language models (LLMs) for qualitative coding often rely on ad hoc prompting with limited guidance for reliability and human oversight. To address these limitations, we propose the Smart Coder Framework, a three-phase approach structuring human-LLM collaborative coding through Instruction, Prompt Strategy, and Reliability Check. We evaluated the framework using 72 middle-school mathematics teacher interview transcripts and a human-consensus benchmark. A factorial design compared instructed versus uninstructed prompts, few-shot versus few-shot plus chain-of-thought prompting, and single-pass versus self-consistency coding. Twelve transcripts were used for prompt development and calibration, and 60 for the formal evaluation. In the results, instruction consistently improved agreement. Chain-of-thought further improved agreement under adequate instruction, and self-consistency produced smaller, theme-dependent gains but helped identify unstable decisions for human review. Performance also varied by transcript characteristics. Overall, the framework systematically structures and monitors LLM-assisted qualitative coding while preserving human authority over coding decisions.
| Comments: | 29 pages, including supplemental materials |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.33205 [cs.AI] |
| (or arXiv:2610.33205v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33205 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Cheng Tang [view email]
[v1]
Fri, 23 Oct 2026 18:52:09 UTC (830 KB)
来源：arXiv cs.AI · arxiv.org