arXiv cs.AI· Addi Malviya-Thakur, Reed Milewicz, Gregory R. Watson, Audris Mockus·· 10 小时前AI 评分24
研究软件指标全景综述:AI 时代如何衡量研究软件的质量与影响力
Measuring What Matters in the Age of AI: The Research Software Metrics Landscape
AI 导读
一项基于 49 篇参考文献的研究软件指标综述将现有指标归为八大类,并按数据来源、所声称的构念、效度证据、自动化潜力和误用风险逐项描述。作者指出,许多被广泛引用的研究软件指标只是部分信号,单独衡量科学影响力时效力薄弱。AI 辅助开发进一步扩大了代码溯源、生成代码可维护性、审稿人投入、模型与数据依赖以及人类劳动署名等方面的缺口。
正文 · 原文
Abstract:Research software now serves as a primary instrument of science, but the methods the community uses to judge its quality, sustainability, reuse, and impact remain scattered across software engineering, open science, FAIR software, citation, and research-assessment literature. AI-assisted development sharpens the problem: code can be produced at rates that outpace human review capacity, provenance and attribution are harder to track, scholarly credit systems cannot keep pace, and existing metrics inherit assumptions about how research software gets developed that may no longer apply. To aid in navigating these challenges, we present a focused landscape synthesis of research software metrics based on 49 substantive references, organizing the recurring metrics into eight families described by data source, claimed construct, evidence of validity, automation potential, and susceptibility to misuse. Many widely cited research software metrics serve as partial signals but make weak standalone measures of scientific impact. The AI era opens or widens gaps around provenance, maintainability of generated code, reviewer effort, model and data dependencies, reuse and community-health signal inflation, and credit for human labor that AI does not perform. We argue for responsible, contextual, mixed-method use of research software metrics, and offer decision guidance for Research Software Engineers (RSEs), principal investigators, institutions, and funders selecting metrics that support better software, fairer credit, and more trustworthy science. All findings rest on the 49-reference focused review; a reproducible search-and-triage protocol and a 727-record triaged corpus accompany the paper as a transparency artifact for a future full systematic mapping study.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computational Engineering, Finance, and Science (cs.CE); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.23253 [cs.SE] |
| (or arXiv:2611.23253v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23253 arXiv-issued DOI via DataCite |
Submission history
From: Addi Malviya Thakur [view email]
[v1]
Fri, 25 Sep 2026 18:48:56 UTC (50 KB)
来源:arXiv cs.AI · arxiv.org