跳到正文
arXiv cs.AI· Matt Turk, Brandon L. Neel, Diana M. Lopez, Zachary D. Smith, Grace N. Johnecheck, James B. Golden·· 10 小时前AI 评分38

ProteinGeoBench:前沿结构生物学智能体擅长查事实,却在几何计算上失分

ProteinGeoBench: Frontier Structural-Biology Agents Are Fluent at Lookup but Lost in Geometry

AI 导读

ProteinGeoBench 用 131 个任务评测智能体的蛋白质结构能力,四个前沿模型各跑多次共 4,249 次运行,另有 26 个模型参与子集测试。Claude Opus 5 查事实得分 0.907、从原子坐标计算几何仅 0.716,Gemini 3.1 Pro 是唯一例外;12 个全部模型失败的任务中有一半是计算题,而计算题仅占任务集的五分之一。

正文 · 原文

View PDF HTML (experimental)

Abstract:ProteinGeoBench is a set of 131 tasks that give an agent protein structure files and ask it to look up recorded facts, check sequences, calculate geometry from atom coordinates or reason about structures. We ran four frontier models on every task several times (4,249 runs) and 26 models on a subset. A single overall score hides two different skills. Three of the four frontier models are best at looking up facts and worst at calculating from coordinates: Claude Opus 5 scores 0.907 on lookup and 0.716 on calculation. Gemini 3.1 Pro is the exception. Geometry is overrepresented among the hardest tasks. Half of the twelve tasks that all four models fail are calculations, although calculations are a fifth of the suite, and eight of the twelve were solved by none of the 26 models. The three leading models are within 0.031 of one another, too close to order, and with one run per task a different model ranks first in 24% of bootstrap resamples. Reports should therefore give scores by task type and the number of runs per task. They should also state the agent harness, the software that runs the model, and its network access, because both change scores. On a 15-task subset, blocking eight structural databases lowered scores, and removing network access lowered them more. Replacing our harness with each provider's own command-line agent moved scores by up to 0.09 in either direction. Across the 26 models, those that scored higher on the other tasks also scored higher on calculation.
Comments: 24 pages, 2 figures, 10 tables. Benchmark and tasks: this https URL
Subjects: Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Biomolecules (q-bio.BM)
Cite as: arXiv:2610.26829 [cs.AI]
  (or arXiv:2610.26829v2 [cs.AI] for this version)
  https://doi.org/10.48550/arXiv.2610.26829

arXiv-issued DOI via DataCite

Submission history

From: Matthew Turk [view email]
[v1] Mon, 19 Oct 2026 17:55:38 UTC (63 KB)
[v2] Wed, 18 Nov 2026 02:39:21 UTC (63 KB)

来源:arXiv cs.AI · arxiv.org