arXiv cs.CL· Zhaoxin Fan, Zheng Zhu, Yuqing Ma, Faguo Wu, Jian Yang, Wenjun Wu·· 12 小时前AI 评分34
Continuum-LM：面向 AI4Science 任务的连续深度大语言模型
Dynamic System Inspired Continuous Large Language Model for AI4Science Tasks
AI 导读
Continuum-LM 将预训练残差变换复用为连续深度语言模型中的动力学算子，隐藏状态通过高阶数值积分求解的常微分方程演化，由求解器阶数和步长控制计算量。在八项科学任务上，该模型在 0.6B 和 1.7B 规模下平均精确匹配准确率分别比离散对应模型提升 7.77 和 10.96 个百分点。0.6B 规模下，它在参数与残差函数评估预算匹配时也优于 Ouro 式循环控制。
正文 · 原文
Abstract:Scientific foundation models increasingly operate on continuous phenomena, yet still use discrete residual computation. We ask whether pretrained residual transformations can support additional latent computation without expanding the backbone. To test this possibility, we introduce Continuum-LM, which reuses these transformations as dynamical operators in a continuous-depth language model. Its hidden states evolve through an ordinary differential equation evaluated by higher-order numerical integration, with solver order and step size controlling computation. Across eight scientific tasks, Continuum-LM trains stably and improves mean exact-match accuracy by 7.77 and 10.96 percentage points over discrete counterparts at the 0.6B and 1.7B scales, respectively. At 0.6B, it also outperforms an Ouro-style recurrent control under matched parameter and residual-function evaluation budgets. Together, these results support extensible computational depth through parameter reuse, identifying continuous hidden-state evolution as a practical design dimension for scientific language models beyond parameter scaling.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.26959 [cs.LG] |
| (or arXiv:2610.26959v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.26959 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jian Yang [view email]
[v1]
Sun, 18 Oct 2026 12:00:44 UTC (2,162 KB)
来源：arXiv cs.CL · arxiv.org