arXiv cs.AI· Guanyu Cai, Ruiming Tian, Lang Yang, Zhouhong Ren, Jinliang Yuan, Lingkun Li, Jiliang Wang·· 12 小时前AI 评分38
你的 NPU 准备好跑 LLM 了吗？拆解移动端 LLM 推理中被忽视的效率瓶颈
Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
AI 导读
一项跨层测量研究显示，移动端 LLM 推理中框架差异在 NPU 上被放大至最高 10 倍，源于卸载策略、算子利用不足与激活量化差异。研究还发现明显的“阶段分化”：NPU 擅长计算密集的 prefill，CPU 则在内存密集的 decoding 上优于其他后端。仅调整 CPU 频率即可降低最高 39.7% 能耗，面向能耗的配置在 NPU 后端三个数据集上可降低最高 53.2% 能耗。
正文 · 原文
Abstract:Deploying Large Language Models (LLMs) on mobile devices enhances privacy and reduces latency, but is severely constrained by hardware inefficiencies. We present the first comprehensive, cross-layer measurement study of mobile LLM inference, spanning five mainstream frameworks (e.g., this http URL and GENIE) and three hardware backends (CPU, GPU, and NPU). To enable this analysis, we develop PowerBench, a fine-grained profiling tool that measures LLM throughput and provides the first backend-specific energy attribution for mobile LLM inference, going beyond traditional device-level measurements. We further conduct hardware microbenchmarks to characterize the memory-system performance and compute capabilities of mobile CPUs, GPUs, and NPUs. Our study yields three critical insights: (1) Framework-induced performance gaps are substantially amplified on NPUs, reaching up to $10\times$ due to differences in offloading strategies, hardware-underutilizing operators, and activation quantization strategies. (2) We identify a distinct ``phase split,'' in which NPUs excel at compute-bound prefilling, while CPUs outperform all other backends in memory-bound decoding. This split is driven by NPUs' preference for large, fixed-shape workloads, which conflicts with the small-kernel, dynamic nature of decoding. (3) Backend-specific profiling uncovers substantial scheduling headroom missed by prior work. Suboptimal thread configurations, uncoordinated NPU sleep latencies, and CPU polling intervals result in unnecessary energy consumption. For example, tuning CPU frequency alone reduces energy consumption by up to $39.7\%$. Leveraging these findings, we estimate that an energy-oriented configuration can reduce energy consumption by up to $53.2\%$ on the NPU backend across three datasets, highlighting the substantial potential for further efficiency improvements in mobile LLM deployment.
| Comments: | Accepted by ACM SIGMETRICS 2027 |
| Subjects: | Hardware Architecture (cs.AR); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.05475 [cs.AR] |
| (or arXiv:2607.05475v2 [cs.AR] for this version) | |
| https://doi.org/10.48550/arXiv.2607.05475 arXiv-issued DOI via DataCite |
Submission history
From: Guanyu Cai [view email]
[v1]
Mon, 6 Jul 2026 09:42:34 UTC (870 KB)
[v2]
Fri, 16 Oct 2026 07:18:47 UTC (1,367 KB)
来源：arXiv cs.AI · arxiv.org