arXiv cs.CL· Yunao Zheng, Bin Wen, Xiaojie Wang, Kaiyu Jiang, Xuanyu Zheng, Changyi Liu, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Jiankang Chen, Xu Wang, Tingting Gao, Han Li·· 2 天前AI 评分37
GSM：用共享全局状态实现高效语言建模
GSM: Efficient Language Modeling with Shared Global State
AI 导读
研究者提出因果编码器-解码器架构 Global State Model（GSM），将历史信息获取与深层计算分离：编码器通过多阶段稀疏检索把长程信息聚合为固定大小的共享状态，解码器层只在该状态上做连续变换，不再重复访问历史。
正文 · 原文
Authors:Yunao Zheng, Bin Wen, Xiaojie Wang, Kaiyu Jiang, Xuanyu Zheng, Changyi Liu, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Jiankang Chen, Xu Wang, Tingting Gao, Han Li
Abstract:Improving language model efficiency requires not only cheaper operations but also eliminating redundant computation. We argue that effective modeling requires information accessibility and computational depth, but not repeated historical retrieval across layers. Guided by this principle, we introduce the Global State Model (GSM), a causal encoder--decoder architecture that separates historical information acquisition from deep computation. The encoder progressively aggregates long-range information through multi-stage sparse retrieval into a fixed-size shared state, while decoder layers perform successive transformations over this state without repeated historical access. This design makes decoder attention and cache costs independent of history length, while dependency-aware pruning further reduces prefill computation. Experiments across multiple model scales show that GSM achieves downstream performance comparable to matched DeepSeek-V4.1-Flash-based baselines, with improvements at some scales, while preserving long-context retrieval capabilities. GSM also improves training and inference throughput and reduces cache overhead. These results demonstrate that eliminating architectural redundancy offers an effective path toward efficient language modeling without sacrificing information accessibility or computational depth.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2609.33465 [cs.CL] |
| (or arXiv:2609.33465v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.33465 arXiv-issued DOI via DataCite |
Submission history
From: Yunao Zheng [view email]
[v1]
Sun, 27 Sep 2026 11:19:21 UTC (1,503 KB)
[v2]
Tue, 13 Oct 2026 03:26:20 UTC (1,538 KB)
[v3]
Thu, 15 Oct 2026 03:28:02 UTC (1,479 KB)
来源：arXiv cs.CL · arxiv.org