arXiv cs.AI· Humza Nusrat, Kundan Thind·· 13 小时前AI 评分42
OpenGray：评估放疗规划智能体决策能力的基准
OpenGray: a benchmark for evaluating the decisions of radiotherapy planning agents
AI 导读
研究团队构建了 OpenGray 基准，在 OpenKBP-Opt 头颈数据集上加入已知正确答案的规划任务，用于评估 LLM 放疗规划智能体的决策。在 30 名患者上测试四个系统，使用 Terra 时 LLM 审查加脚本与完整智能体的决策准确率分别为 95.4% 和 92.1%，但对需要解剖学算术的重叠矛盾仅升级 50% 和 25%。
正文 · 原文
Abstract:Background and purpose: Radiotherapy planning agents based on large language models (LLMs) decide when to submit a plan or request review but are evaluated mainly by dose. This study developed OpenGray, a benchmark of planning tasks with known correct decisions. Materials and methods: OpenGray adds tasks with known answers to the OpenKBP-Opt head-and-neck dataset, including two contradictions with dose goals proven incompatible and a feasible control with a stored plan meeting every hard goal. Four systems, from a rule-based planning script to a full LLM agent, were evaluated on 30 patients, the LLM-based ones with a commercial (Terra) and a local (Ministral 3 8B) LLM. Decisions, achieved dose, problem recognition, and technical failures were recorded and also read from logs of 39 stereotactic radiosurgery runs of SAGE, a clinical LLM agent. Results: With Terra, the script preceded by an LLM review and the full agent had decision accuracies of 95.4% and 92.1%, respectively, yet escalated only 50% (95% confidence interval, 25-75%) and 25% (10-40%) of overlap contradictions, which required arithmetic on the anatomy. All four systems submitted every feasible control, but 24% of these plans missed a hard goal that the stored plan had met. In four SAGE runs, one goal failed at every attempt; the agent identified it in every reply but had no action for stopping. Conclusions: Favorable aggregate scores concealed specific decision failures. Planning agents should be evaluated on their decisions as well as their dose and given an explicit action for stopping and requesting review.
| Comments: | 25 pages, 4 figures, 1 table, and the supplementary material (19 tables, 5 figures). Ancillary files: the paper in Markdown, structured results with the sentence each comes from, and an this http URL index for AI agents |
| Subjects: | Artificial Intelligence (cs.AI); Medical Physics (physics.med-ph) |
| Cite as: | arXiv:2610.37081 [cs.AI] |
| (or arXiv:2610.37081v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.37081 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Humza Nusrat [view email]
[v1]
Tue, 27 Oct 2026 11:42:47 UTC (1,682 KB)
来源：arXiv cs.AI · arxiv.org