arXiv cs.CY· Yuan An·· 2 天前AI 评分22
LLM 智能体如何编排科研流程：MCQ 生成与评估的试点研究
Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluation
AI 导读
一项混合方法试点研究让人类研究者协调多个 LLM 智能体，完成从 PDF 提取题目、将开放教材转为结构化表示、对齐教材内容到生成并评估 MCQ 的全流程。研究收集 1,071 道 SAT Math MCQ，用 24 项质量标准评估，生成题目平均质量较高，但从未达到 24/24 全部等效，差距集中在技能深度、认知投入、难度校准和元数据对齐。研究者工作也从"编写题目"转向规格制定、编排、验证与治理。
正文 · 原文
Abstract:Advances in large language models (LLMs) are rapidly transforming scientific work, yet empirical evidence on how these systems reshape research activities remains limited. We report a mixed-methods pilot evaluation of an AI-orchestrated research workflow in which a human researcher coordinated multiple LLM-based agents to perform data extraction, corpus construction, artifact generation, and artifact evaluation. Using the generation and assessment of multiple-choice questions (MCQs) as a testbed, we collected 1,071 SAT Math MCQs and employed LLM agents to extract questions from PDFs, retrieve and convert open textbooks into structured representations, align each MCQ with relevant textbook content, generate new MCQs under specified difficulty and cognitive levels, and evaluate both original and generated MCQs using a 24-criterion quality framework. Across all evaluations, average MCQ quality was high. However, criterion-level analysis and equivalence testing show that generated MCQs are not fully comparable to expert-vetted baseline questions. Strict similarity (24/24 criteria equivalent) was never achieved. Persistent gaps concentrated in skill\ depth, cognitive engagement, difficulty calibration, and metadata alignment, while surface-level qualities, such as {grammar fluency}, {clarity options}, {no duplicates}, were consistently strong. Beyond MCQ outcomes, the study documents a labor shift. The researcher's work moved from ``authoring items'' toward {specification, orchestration, verification}, and {governance}. Formalizing constraints, designing rubrics, building validation loops, recovering from tool failures, and auditing provenance constituted the primary activities. We discuss implications for the future of scientific work, including emerging ``AI research operations'' skills required for AI-empowered research pipelines.
| Subjects: | Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC) |
| Cite as: | arXiv:2602.18891 [cs.CY] |
| (or arXiv:2602.18891v2 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2602.18891 arXiv-issued DOI via DataCite |
Submission history
From: Yuan An [view email]
[v1]
Sat, 21 Feb 2026 16:20:07 UTC (1,731 KB)
[v2]
Thu, 15 Oct 2026 14:47:54 UTC (841 KB)
来源：arXiv cs.CY · arxiv.org