LARC：面向冻结模型学习的低秩自适应残差连接

View PDF HTML (experimental)

Abstract:Low-Rank Adaptive Residual Connections (LARC) give a frozen model a compact numerical state that can learn from feedback. The map $h+BAh$ changes the hidden input to a pretrained mixer. An episode copies a learned initialization $\\rho$ into a private state $\\Phi$, updates its factors when feedback arrives, and resets to the bound $\\rho$. We analyze the factor-space updates and trace this input-side policy carrier into Orbit, the post-training realization of Memory-Mediated Learning Architecture studied here. In Orbit's released default, the learned residual shapes predictions while memory adaptation changes the retained evidence.\n\nThe controlled studies use a rank-4 residual with 12,288 factor entries on frozen MiniCPM5-1B-SFT. Two feedback updates reduce expected query execution error by 24.65 and 36.65 percentage points relative to the respective trained reset states in a four-candidate development task with 16 parameter groups and three paired seeds. A direct support-loss rule reaches 0.78125\\% error. In repository-balanced chronological replay of public continuous-integration jobs, retaining updates raises half-Brier loss from 0.1274 to 0.1808. A follow-up intervention observes same-batch non-descent and inconsistent future benefit from shrinking steps. These comparisons measure when numerical state helps a later read and when the trained reference is more effective.

Comments:	20 pages, 7 figures, 15 tables. Technical report of MMLA. The authors contributed equally. Version 2 adds the Orbit realization, a factor-state counterexample, and updated public component access
Subjects:	Machine Learning (cs.LG); Computation and Language (cs.CL)
Cite as:	arXiv:2609.40063 [cs.LG]
	(or arXiv:2609.40063v2 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2609.40063 arXiv-issued DOI via DataCite

Submission history

From: Junyi Zou [view email]
[v1] Wed, 30 Sep 2026 16:21:39 UTC (463 KB)
[v2] Tue, 10 Nov 2026 00:57:38 UTC (517 KB)