arXiv cs.CL· Esther Shizgal, Ron Keinan, Daniel Shtauber, Roee Tzvi Esquira, Ori Aronson, Matan Toledano, Or Rappel-Kroyzer, Liron Lavi, Renana Keydar, Ittai Bar-Siman-Tov, Reut Tsarfaty·· 12 小时前AI 评分24
从 DQI 到 C-DQI：用于衡量计算话语质量指数的自动码本生成与评估
From DQI to C-DQI: Automatic Codebook Generation and Evaluation for Measuring Computational Discourse Quality Index
AI 导读
研究提出一套基于 LLM 的码本生成与评估完整协议，从话语质量指数（DQI）出发，对比自上而下（理论驱动）、自下而上（数据驱动）与混合三种方式生成适配上下文的评估框架。在 100 份以色列议会记录上的实验显示，归纳式数据驱动码本在保持有效性的同时，模型间一致性（IMA）高于自上而下方法。
正文 · 原文
Authors:Esther Shizgal, Ron Keinan, Daniel Shtauber, Roee Tzvi Esquira, Ori Aronson, Matan Toledano, Or Rappel-Kroyzer, Liron Lavi, Renana Keydar, Ittai Bar-Siman-Tov, Reut Tsarfaty
Abstract:Effective democratic deliberation is vital for civic engagement and the health of legislative institutions. However, assessing parliamentary discourse quality is traditionally done using expert-scored, theory-based codebooks (CBs), which are rigid, labor-intensive, and prone to biases. While Large Language Models (LLMs) offer a scalable alternative for both schema induction and discourse annotation, ensuring schema quality and mitigating models' inherent bias remains a clear challenge. In this work, we propose a complete protocol for LLM-based codebook generation and evaluation. Starting from the widely accepted discourse quality Index (DQI), we compare top-down (theory-based), bottom-up (data-driven), and hybrid approaches for generating context-adapted schemas. To evaluate these schemas, we define an extrinsic proxy measure, validated by manual annotations, alongside an intrinsic inter-model agreement (IMA) metric, which leverages differing model biases to signal overall reliability. Our experiments on 100 Israeli parliament transcripts reveal that inductive, data-driven schemas achieve higher IMA than top-down methods while maintaining validity, demonstrating their potential for reliable, large-scale analysis of political discourse.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.27937 [cs.CL] |
| (or arXiv:2610.27937v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.27937 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ron Keinan [view email]
[v1]
Tue, 20 Oct 2026 09:45:13 UTC (539 KB)
来源：arXiv cs.CL · arxiv.org