arXiv cs.CL· Fumiki Yohena, Toshiaki Sota, Shota Suzuki, Tomoko Okamoto, Michiaki Tatsubori·· 10 小时前AI 评分31
用仓库渲染参考图提升编码智能体的图像理解能力
Repository-Context-Guided Image Understanding for Agentic Software Development
AI 导读
研究提出为编码智能体补充由同一代码库渲染的清晰参考截图，帮助其在低分辨率、压缩失真或小字号文本的 UI 截图中定位相关界面元素与源文件。在基于 SWE-bench Multimodal 构建、向真实 Web 应用注入 UI 视觉缺陷的受控基准上，参考图在四倍下采样加 JPEG 压缩条件下提升了 gold file 匹配率，识别文本的准确率比语言偏置 OCR 高约 12 个百分点。
正文 · 原文
Abstract:Coding agents increasingly receive UI screenshots as part of bug reports, yet degraded image quality - caused by low resolution, compression artifacts, or small rendered text - can prevent them from identifying the relevant UI elements and source files. Vision-language models become less reliable as image quality deteriorates, while OCR-assisted approaches may propagate recognition errors from degraded inputs. Both approaches are limited when the visual evidence required for localization is no longer clearly observable in the query screenshot. We propose augmenting coding agents with repository-rendered reference images: crisp screenshots generated from the same codebase that the agent is tasked with modifying. These references provide readable visual evidence of the application's UI while preserving the current, potentially bug-containing implementation state. Because they are generated from the available repository rather than requiring a known bug-free version, they can be produced automatically for reproducible application states without manual image annotation. We evaluate the approach on a controlled benchmark derived from SWE-bench Multimodal by injecting visual UI bugs into a real web application. We vary screenshot degradation and reference-image provision and measure whether the coding agent identifies the correct source file. The results show that reference images improve file localization especially for degraded image quality, increasing the gold file match rate under fourfold downsampling with JPEG compression. We confirmed with further analysis that the proposed approach recognizes text approximately 12 percentage points better than language biased OCR under the same degradation.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.13026 [cs.SE] |
| (or arXiv:2611.13026v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13026 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Michiaki Tatsubori [view email]
[v1]
Mon, 9 Nov 2026 08:33:52 UTC (3,795 KB)
来源：arXiv cs.CL · arxiv.org