arXiv cs.CL· Saurav Bellam (The University of Texas at Austin)·· 14 小时前AI 评分42
小型编码智能体如何使用代码图与嵌入向量？
How Do Small Coding Agents Use Code Graphs and Embeddings?
AI 导读
研究在 SWE-bench Verified 上测试小型开源模型使用嵌入搜索与代码图工具的效果：结构化工具让 Qwen3-8B 和 Gemma 4 E4B 的定位准确率翻倍，增益在 1.7-4B 规模达 17-21 分，到 32B 降至 4-5 分。
正文 · 原文
Authors:Saurav Bellam (The University of Texas at Austin)
Abstract:Coding agents are given embedding search and code-graph tools on the premise that a repository's structure is something the model lacks and the tool supplies. We test that premise on SWE-bench Verified with small open-weight models, stratifying tasks by a deterministic hop depth (the shortest code-graph path from code the issue mentions to code the gold patch changes) that a companion paper defines and validates. First, we ablate a localization agent's tool set (grep only; +embedding search; +graph tools; all) on a depth-balanced 160-task split and on all 500 tasks for ten models (six Qwen3 sizes from 0.6B to 32B with five decodings each, an FP8 control, three Gemma 4 sizes; 79k episodes). Structured tools double grep-only accuracy for Qwen3-8B and Gemma 4 E4B; their gain peaks at 17-21 points at 1.7-4B and falls to 4-5 points at 32B (odds ratio 0.77-0.82 per doubling of parameters). From 8B up, about half of that gain is the form of the submission rather than finding the target; the fall from 14B to 32B is a fall in tool uptake at equal grep-only accuracy; and the same 4B size gains or does not gain from a tool depending on the checkpoint. Each extra hop multiplies the odds of correct localization by 0.50-0.67 at every size, and no tool set flattens that gradient: traces show the graph tools' gain is name lookup at hop 0 and one expansion at hop 1, and agents chain expansions in only 12-14% of episodes. A fixed repair model resolves 6-13 of 160 tasks from these localizations (oracle 26). Second, linear probes find direct call edges decodable under a within-anchor, degree-stratified protocol (AUC 0.74; 0.61-0.66 inside agent conversations), but below a lexical baseline (0.99), with two-hop edges at 0.57. Small agents carry a one-hop, largely lexical picture of their repository, in weights and in use. We release the agent, probes, logs and traces.
| Comments: | 29 pages, 5 figures. Depth labels and dataset card and full code, raw results and analysis will be released in the near future |
| Subjects: | Software Engineering (cs.SE); Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.33958 [cs.SE] |
| (or arXiv:2610.33958v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33958 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Saurav Bellam [view email]
[v1]
Sun, 25 Oct 2026 00:16:32 UTC (197 KB)
来源：arXiv cs.CL · arxiv.org