arXiv cs.CY· Shiv Muthupandiyan, Tolga H. Dur, Mac Walker, Timo Flesch, Mahmoud Ghanem, Steph Suddell, Giles Harper-Donnelly, Sophie Rose, Jasper G\"otting, Coleman Breen, Jake Pencharz·· 3 天前AI 评分40
ReproBAIT：编程智能体能否复现自己的生物 AI 工具？
ReproBAIT: Can coding agents build their own biological AI tools?
AI 导读
研究团队提出 ReproBAIT 评测，在无联网的沙箱中让编程智能体仅凭有限描述、固定时间与算力预算复现已发表的生物 AI 工具，并以留出数据上的工具专属指标打分。
正文 · 原文
Authors:Shiv Muthupandiyan, Tolga H. Dur, Mac Walker, Timo Flesch, Mahmoud Ghanem, Steph Suddell, Giles Harper-Donnelly, Sophie Rose, Jasper Götting, Coleman Breen, Jake Pencharz
Abstract:Large language models are increasingly competent at leveraging scientific knowledge and existing computational tools to execute complex workflows. At present, little information exists on their ability to reconstruct tools when given limited descriptions of such tools. We introduce an evaluation titled ReproBAIT (Reproducing Biological AI Tools), which measures the ability of coding agents to design, implement, and train biological AI tools taken from published research across a range of information access conditions. The coding agent is securely sandboxed without internet access and receives training data, a fixed time and compute budget, and limited information about the tool to replicate. We then score the reproduced biological AI tool based on its performance (tool-specific metrics) on held-out data. We evaluated a set of biological AI tools of varying complexities: ProteinMPNN, an inverse protein folding tool, RFdiffusion3, an all-atom generative diffusion tool, and a closed-source molecular design tool. We institute a methodology to check for memorisation, as the open source tools are likely to be included in models' pretraining corpus. Across a range of leading closed- and open-source coding agents, we find substantial reproduction capability for some selected tools, though limited capability for a highly complex tool. We find moderate evidence consistent with memorisation, varying across coding agents. These results suggest that tool developers need to be increasingly cautious when releasing information about the tools publicly; re-creation of the tool based on such information could undermine a tool's commercial value or invalidate dual-use managed access strategies.
| Subjects: | Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19810 [cs.AI] |
| (or arXiv:2610.19810v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19810 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tolga H. Dur [view email]
[v1]
Wed, 14 Oct 2026 08:50:11 UTC (170 KB)
来源：arXiv cs.CY · arxiv.org