TensorDex：面向现代 AI 模型的紧凑张量中心存储系统

View PDF HTML (experimental)

Abstract:Modern model hubs store hundreds of petabytes of large language models (LLMs), with fine-tuned variants dominating the storage footprint. These variants contain substantial cross-model redundancy that delta compression can exploit by storing only the difference between a target and a reference model. However, compression effectiveness depends critically on choosing a similar reference. At model-hub scale, this is challenging because model lineage metadata is often missing or unreliable, and different tensors within the same model may be most similar to tensors from different models. Consequently, model-level pairing leaves substantial redundancy unexploited.
We present TensorDex, a lossless, tensor-centric storage system that performs delta compression at tensor granularity. TensorDex decomposes models into tensors, uses compact bit-level fingerprints to predict tensor-pair compressibility, and incrementally organizes tensors into multi-center clusters to select effective bases as new models arrive. Evaluated on 2,890 randomly sampled Hugging Face models, TensorDex reduces storage footprint by 70.5%, 37% lower than the state-of-the-art design. It achieves 22.9 GB/s compression and 28.4 GB/s decompression throughput, 3.86x and 1.49x faster than the next-best system, respectively.

Comments:	12 pages, 6 figures. Systems paper on AI model storage
Subjects:	Distributed, Parallel, and Cluster Computing (cs.DC); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
ACM classes:	D.4.2; E.5; I.2.6
Cite as:	arXiv:2604.17104 [cs.DC]
	(or arXiv:2604.17104v3 [cs.DC] for this version)
	https://doi.org/10.48550/arXiv.2604.17104 arXiv-issued DOI via DataCite

Submission history

From: Tingfeng Lan [view email]
[v1] Sat, 18 Apr 2026 18:40:29 UTC (1,771 KB)
[v2] Tue, 12 May 2026 19:14:25 UTC (1,762 KB)
[v3] Thu, 15 Oct 2026 03:05:30 UTC (1,846 KB)