arXiv cs.CL· Yuan Pu, Yudong Lu, Ting Wang, Zhicong Hong, Qinuo Li, Xuansheng Wu, Na Zou, Xia Hu, Guanchu Wang·· 12 小时前AI 评分48
OpenESM：能量引导模型是可扩展的基础学习器
OpenESM: Energy-Steered Models Are Scalable Foundation Learners
AI 导读
研究者提出能量引导模型（ESM），将能量引导与 token 生成整合进单一架构，先在能量景观上优化隐状态再生成 token。在三个大规模语料上预训练后，ESM 在 26 个下游任务中的多数上优于 GPT，并随模型规模与训练数据增长展现良好扩展性。模型权重与训练基础设施已全部开源。
正文
Abstract:Energy-based models have shown promising properties for domain-specific tasks but remain challenging in modeling general-purpose scenarios. To bridge this gap, we design energy-steered models (ESMs) for foundational language modeling. ESMs integrate energy steering and token generation within a single architecture to leverage the energy-guided search to refine token generation. Specifically, ESMs first optimize a latent state over the energy landscape, then leverages the refined latent state for token generation. By performing such energy-guided pre-training on large-scale corpora, ESMs learn generally effective energy landscape where favorable tokens correspond to lower energy values. To explore the scaling behavior of ESMs, we pre-train ESMs on three large-scale corpora, then evaluate across 26 downstream tasks spanning a broad range of domains. Our results demonstrate that ESMs have promising scaling behaviors with increasing model size and training data, and outperform GPTs on most downstream tasks. These findings highlight ESMs as a promising architecture for large-scale pre-training and general purpose problem solving. We fully release the model weights and training infrastructure to enable the community to completely reproduce and extend our results.
GitHub: this https URL HuggingFace: this https URL
| Comments: | 12 pages, 7 figures, 3 tables |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.33005 [cs.CL] |
| (or arXiv:2610.33005v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.33005 arXiv-issued DOI via DataCite |
Submission history
From: Guanchu Wang [view email]
[v1]
Wed, 7 Oct 2026 13:19:02 UTC (5,625 KB)
来源：arXiv cs.CL · arxiv.org