HydraLLM：面向弹性文本与视觉语言模型的自适应宽度切片架构

View PDF HTML (experimental)

Abstract:LLMs and LVLMs are increasingly deployed across heterogeneous hardware, ranging from data-center GPUs to resource-constrained edge devices. Their fixed-size architectures limit the ability to reuse a single model across deployment settings with different compute, latency, and model-size targets. We introduce HydraLLM, an elastic MQA-based architecture. Its number of active attention heads determines the effective widths of the embeddings, attention projections, feed-forward layers, and the output head, forming a nested family of subnetworks within a single parameter set. We introduce a static-to-elastic conversion pipeline that, starting from a pre-trained model checkpoint, includes importance-driven weight reordering and stochastic subnetwork training with knowledge distillation. We extend this principle to multimodal inference through an elastic projection module whose output width follows the active text width. HydraLLM covers a wider range of active parameter configurations than FFN-only elastic architectures. With 75% of heads active, HydraLLM achieves 2.2 percentage points higher TriviaQA accuracy than a state-of-the-art nested baseline configuration which has 17.25% more parameters. With 50% of heads active, our multimodal model achieves a CLIPScore comparable to that of the 7$\times$ larger Gemma 3 4B model. With only 25% of heads active, HydraLLM increases token-generation throughput by 75.3% over the full-width configuration.

Comments:	Accepted at the BabyLM Workshop at EMNLP 2026
Subjects:	Computation and Language (cs.CL); Machine Learning (cs.LG)
ACM classes:	I.2.7; I.2.6
Cite as:	arXiv:2610.23455 [cs.CL]
	(or arXiv:2610.23455v1 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2610.23455 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Oscar Kemme [view email]
[v1] Fri, 16 Oct 2026 13:38:11 UTC (1,103 KB)