arXiv cs.AI· Honson Ling, Kidus Admassu, Stephanie Rosenthal·· 12 小时前AI 评分29
专家批评结合 LLM 驱动优化：迭代构建高质量 AI 评测数据集的方法
Pairing Expert Critique with LLM-Driven Refinement: An Iterative Approach for Curating Quality AI Evaluation Datasets
AI 导读
研究者提出 SME-C 人机协作流程，通过专家对候选回答的轻量批评结合 LLM 驱动的分析与优化，迭代构建回答数据集和评测标准。在金融商业简报领域，411 位金融专业人士对 520 个查询给出 3,853 条批评；盲测中两种 LLM 优化策略均显著优于专家排名最高的基线回答，且改进集中在原本较弱的回答上。基于批评的优化更贴合领域知识，基于投票的优化更擅长识别并删除无关内容。
正文 · 原文
Abstract:Evaluating and improving domain-specific LLM responses requires high-quality reference data and rubrics, yet expert authoring is slow and expensive, and synthetic datasets may fail to capture practitioner preferences unless they are grounded in expert judgment. We present SME-C (Subject Matter Expert Critique), a human-AI workflow that captures expertise through lightweight critiques of candidate responses and complements it with LLM-driven analysis and refinement to iteratively curate response datasets and evaluation rubrics. We validate SME-C in the domain of financial business briefs, collecting 3,853 critiques from 411 finance professionals on 520 queries across two survey rounds. We evaluate two LLM-driven refinement strategies: one guided by the experts' written critiques and one that infers critique solely from their preference votes. In a blind validation survey with 100 additional professionals, we found that both strategies significantly improved quality over the experts' top-ranked baseline responses. The gains were concentrated among weaker original responses, and the two strategies excelled in different ways: critique-based refinement produced more domain-grounded improvements, whereas vote-based refinement was more effective at identifying and removing irrelevant content. The resulting rubrics and refined dataset support reproducible offline evaluation and benchmarking of LLMs, suggesting that structured expert critique is a viable alternative to expert authoring for building high-quality, domain-specific evaluation resources more quickly, at scale, and at lower cost.
| Comments: | 20 pages, 7 figures, 9 tables |
| Subjects: | Human-Computer Interaction (cs.HC); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| ACM classes: | H.5.3; I.2.7; H.1.2 |
| Cite as: | arXiv:2611.05749 [cs.HC] |
| (or arXiv:2611.05749v1 [cs.HC] for this version) | |
| https://doi.org/10.48550/arXiv.2611.05749 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Honson Ling [view email]
[v1]
Tue, 3 Nov 2026 23:38:56 UTC (2,889 KB)
来源：arXiv cs.AI · arxiv.org