arXiv cs.AI· Mohammad Arvan, Amber E. Osterholt, Bailee Rue, Yuvaneswaren R. Sureshbabu, Krishna R. Patel, Rebecca T. Feinstein, Bethany C. Bray, Niranjan S. Karnik·· 14 小时前AI 评分34
AI Agent 起草转化影响摘要的真实场景评估：81.7% 的发现被接受或编辑
Real-World Evaluation of an AI Agent Drafting Translational Impact Summaries
AI 导读
一个 human-in-the-loop AI Agent 在 CTSA 影响报告流程中为 10 名 KL2/K12 学者起草 TSBM 影响摘要，两名评审对 507 条发现中 81.7% 予以接受或编辑。评审每人每名学者中位耗时 14 分钟，替代约 15 小时的人工整理，综合准确性评分 4.5、实用性 4.8（5 分制）。约三分之一的发现属于常规流程易漏的非学术类别。
正文 · 原文
Abstract:Introduction. Clinical and Translational Science Award (CTSA) programs must document their scholars' research impact, but assembling each scholar's record by hand takes staff an estimated 15 hours and does not scale to a full cohort. An artificial intelligence (AI) agent could serve as a tool to gather scholar data across platforms and disciplines. Methods. We built a human-in-the-loop AI agent that assembles a dossier of sourced evidence for each scholar and drafts one-sentence Translational Science Benefits Model (TSBM) impact summaries for staff review. We evaluated it in the impact-reporting workflow of one CTSA hub across 10 career-development (KL2/K12) scholars. Two evaluation staff independently coded all 507 findings as accept, edit, or reject; the primary measure was the unanimous usable rate, defined as the share both accepted or edited. Results. Both reviewers accepted or edited 81.7% of the agent's findings. Reviewers each spent a median of 14 minutes per scholar, replacing an estimated 15 hours of manual assembly. Inter-rater agreement was moderate (Cohen's kappa 0.43 on the usable-versus-reject decision). A profile discovery study found the agent's recall close to human search. The agent's impact evidence spanned all four TSBM domains, and about a third of the reviewed findings fell in non-scholarly categories that routine processes tend to miss. Reviewers rated synthesis accuracy 4.5 and usefulness 4.8 on a 5-point scale. Conclusions. A human-in-the-loop AI agent can serve as the first-pass author of a scholar's impact record, shifting staff from collecting and writing to reviewing, and making cohort-scale impact reporting feasible.
| Comments: | 15 pages, 5 figures, 2 tables. Submitted to the Journal of Clinical and Translational Science. Code: this https URL |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Digital Libraries (cs.DL); Human-Computer Interaction (cs.HC) |
| ACM classes: | I.2.7; I.2.11; H.3.7; H.5.2 |
| Cite as: | arXiv:2607.16989 [cs.CL] |
| (or arXiv:2607.16989v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.16989 arXiv-issued DOI via DataCite |
Submission history
From: Mohammad Arvan [view email]
[v1]
Sat, 18 Jul 2026 22:34:52 UTC (205 KB)
[v2]
Tue, 21 Jul 2026 15:50:03 UTC (204 KB)
[v3]
Thu, 29 Oct 2026 16:48:04 UTC (232 KB)
来源：arXiv cs.AI · arxiv.org