arXiv cs.CL· Areej Mehboob, Muhammad Ali Shafique, Muhammad Usman Qadeer, Hamza Farooq·· 11 小时前AI 评分32
量化如何影响编码智能体：不止于准确率
Beyond Accuracy: How Quantization Affects Coding Agents
AI 导读
研究对比 BF16 与 4-bit AWQ、GPTQ 量化后的 Qwen2.5-Coder-Instruct（7B、14B、32B）和 Devstral-Small-2507（24B）在 OJBench 编码任务上的表现，未发现统计显著的准确率损失。
正文 · 原文
Abstract:Agents are increasingly used for coding tasks, but the inference cost of large language models (LLMs) limits where they can run. Quantization reduces model size, but its effects on coding agents that execute and revise programs remain underexplored. We compare BF16 with 4-bit AWQ and GPTQ for Qwen2.5-Coder-Instruct (7B, 14B, and 32B) and Devstral-Small-2507 (24B) on OJBench coding problems across three seeds. We detect no statistically significant accuracy loss for any of the coding agents. However, final accuracy can conceal substantial behavioral changes: AWQ reduces Qwen-7B one-shot solves from 17 to 0 while increasing loop-based solves from 10 to 25. All Qwen-32B successes involve tool use, whereas Devstral rarely makes a valid executor call. Failure profiles also differ: Qwen-32B produces far more compilation errors than the smaller Qwen models, rising slightly under quantization, while Devstral's failures shift significantly from runtime errors toward wrong answers and timeouts. These findings indicate that quantization robustness in coding agents should be assessed through solution trajectories, tool use, and failure modes, in addition to final accuracy.
| Comments: | Accepted at the NeurIPS 2026 Workshop on Small Language Models for Agentic Systems (SLM-Agents), Paris, France |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.02368 [cs.CL] |
| (or arXiv:2611.02368v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.02368 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Muhammad Ali Shafique [view email]
[v1]
Mon, 2 Nov 2026 01:30:27 UTC (26 KB)
来源：arXiv cs.CL · arxiv.org