arXiv cs.CL· Abhinav Sudhakar Dubey, Vidushi Goel, Kiran Rana, Shikhar Mathur, Vaggos Chatziafratis·· 13 小时前AI 评分37
通过自主进化提升 VLM 的科学图表分析能力
Improving VLMs on Scientific Chart Analysis via Autonomous Evolution
AI 导读
研究用自主进化方法搜索推理程序与代码，在不改动模型权重的前提下提升冻结 VLM 的科学图表问答准确率。在三个 VLM 和四个科学图表基准上，描述性问题的精确匹配率平均从 16.6% 升至 41.3%，单项最大提升 55.2 个百分点。进化后的程序还比初始程序快 3.8 倍，表明大量错误源于推理流程而非模型本身。
正文 · 原文
Abstract:Understanding and operating with scientific charts is a standard approach in data analysis, and while humans are good at interpreting such charts, open vision-language models (VLMs) are known to perform poorly when it comes to question answering based on such data types. Motivated by an initial observation that many of their errors are due to inexact answering, we raise the following natural question: how much accuracy can we recover without changing the model at all? In this paper, we study this question by searching over inference programs and the code around a "frozen" version of the model. Using ideas from autonomous evolution (famously used in AlphaEvolve), we first start from a baseline inference program, and in each round we show an LLM judge the code together with a few failures on training questions, and ask for an improving evolutionary move. As usual in autonomous evolution approaches, the candidate programs are kept in a small pool ranked by training accuracy, and the next edit is performed on top-scorers from this pool. We run this approach on three VLMs and four popular benchmarks on scientific charts, and we measure the fraction of descriptive questions whose answer exactly matches the ground truth. The accuracy improves in all experimental settings. On average, it rises from 16.6% with the starting program to 41.3% with the evolved one, and the largest single increase is 55.2 percentage points. As a side benefit, our evolved program is 3.8x faster than the starting program, because short answers are easier to grade and faster to generate. Our results demonstrate that a large part of the observed errors arise due to the inference pipeline, and that optimizing it can improve the "frozen" VLM without the need to retrain it.
| Comments: | 13 pages, 5 figures, 4 tables. Code: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.35077 [cs.CV] |
| (or arXiv:2610.35077v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.35077 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Abhinav Sudhakar Dubey [view email]
[v1]
Mon, 26 Oct 2026 09:22:52 UTC (157 KB)
来源：arXiv cs.CL · arxiv.org