arXiv cs.AI· Bowen Shi, Qi Ouyang, Long Qian·· 12 小时前AI 评分31
稀疏正则化如何让多任务神经网络自发形成功能模块化
Sparsity-driven emergence of functional modularity in multitask neural networks
AI 导读
稀疏正则化能在不预设模块边界的情况下促使多任务神经网络形成功能模块化。研究在多任务多层感知机与轻量 Transformer 中识别出紧凑的任务选择性子网络（functional tokens），选择性保留这些 token 可维持对应任务表现、同时降低其他任务性能，且这种专门化在训练中优先出现在更深的层。
正文 · 原文
Abstract:Neural networks can learn multiple tasks, but how learning organizes shared computation into specialized and reusable components remains poorly understood. In this work, we show that sparsity inducing regularization promotes functional modularity in multitask networks without predefined module boundaries. Using functional profiling in multilayer perceptrons and a lightweight transformer, we identified compact, task-selective subnetworks, termed functional tokens. Across rule-based task families, selectively retaining these tokens preserves their associated functions while reducing performance on other tasks. Such specialization develops preferentially in deeper layers during training, while earlier layers retain shared computation. Regularization strength and form shape module compactness and specialization, while task weighting alters functional allocation. Moreover, extracted functional tokens can be recombined through alternative pathways to reconstruct a learned composite computation without retraining, preserving their connectivity and learned weights. Our results suggest that sparsity-constrained optimization shapes not only parameter use but also the organization and reuse of learned computation.
| Comments: | 34 pages, 6 figures, including Supplementary Information |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.32153 [cs.LG] |
| (or arXiv:2610.32153v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.32153 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Bowen Shi [view email]
[v1]
Fri, 23 Oct 2026 04:38:24 UTC (5,601 KB)
来源：arXiv cs.AI · arxiv.org