arXiv cs.AI· Tomer Galanti, Josh Susskind, Eran Malach·· 3 天前AI 评分41
少比特多尝试：内存受限下如何恢复模型能力
Fewer Bits, More Tries: Recovering Capability Under Memory Constraints
AI 导读
在内存受限的本地部署场景下，用较小或压缩后的模型并行生成多个候选答案、再借助公开或模型自生成的测试筛选，可恢复因模型规模或压缩损失的大部分准确率。7B Qwen 模型八次并行尝试在 MBPP+ 上基本追平 32B 模型，在另外三个代码生成基准上超过它，同时模型内存占用约少 4 倍、速度最高快 6 倍、能耗更低（基于 Nvidia GPU 实测）。
正文 · 原文
Abstract:Large language models are generally more capable than smaller ones, but their memory requirements can make local deployment difficult. We ask whether part of this capability gap can instead be recovered by running a smaller or more compressed model multiple times and selecting among its outputs using public or model-generated tests. Single-user local inference can leave hardware underutilized, allowing parallel candidate generation to trade additional test-time computation for accuracy at modest latency cost. Across four code-generation benchmarks, verified retry recovers substantial accuracy lost to model size or compression. A 7B Qwen model with eight parallel attempts essentially matches a 32B model on MBPP+ and exceeds it on the other three benchmarks, while using about $4\times$ less model memory, running up to $6\times$ faster, and using less energy, as measured on Nvidia GPU. We observe similar tradeoff across model sizes, compression levels and hardware choices, although exact benefit depends on the hardware profile and problem setting. These results show that model capacity can be traded for verified test-time computation when correct solutions remain reachable within a modest retry budget.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15372 [cs.AI] |
| (or arXiv:2610.15372v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15372 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tomer Galanti [view email]
[v1]
Sun, 11 Oct 2026 05:52:48 UTC (292 KB)
来源：arXiv cs.AI · arxiv.org