arXiv cs.CL· Ali Sarabadani, Massimo Piccardi, Mehran Abolhasan, Farzad Tofigh, Ian Zhou, Justin Lipman·· 12 小时前AI 评分24
FederatedBytes：面向高效多模态联邦学习的无分词器字节级小语言模型
FederatedBytes: Tokenizer-Free, Byte-Level Small Language Models for Efficient Multimodal Federated Learning
AI 导读
FederatedBytes 是一个无分词器的字节级小语言模型，通过将文本、图像等输入统一视为字节序列并结合线性注意力机制，降低联邦学习的内存与计算开销。在多语言和多模态情感分析任务上，该模型基于 Multilingual Amazon Reviews 和 ScienceQA 数据集评测，准确率超过 ByT5 和 LSTM 等对比模型，同时显著减少内存占用和通信开销。
正文 · 原文
Abstract:Federated Learning (FL) is a powerful machine learning framework that allows building a global model from the contributions of multiple, distributed clients, while ensuring the privacy of their local data. However, its heavy computational and communication overheads challenge its use for contemporary AI workloads such as natural language processing (NLP) and multimodal analysis. For this reason, this paper presents a novel small language model that leverages byte-level text and multimodal data processing, enhanced by a linear attention mechanism, to enable manageable Federated Learning of NLP and multimodal analysis tasks. The proposed model, nicknamed FederatedBytes, manages to process text and any other data types, including images, by treating its input as a mere sequence of bytes, while its lean linear attention mechanism allows abating the overall memory and computational footprint. To prove the effectiveness of the proposed model we have carried out an extensive performance evaluation on sentiment analysis tasks in multilingual and multimodal settings over the Multilingual Amazon Reviews and ScienceQA datasets. The experimental results show that FederatedBytes has been able to outperform comparable models such as ByT5 and LSTM, achieving higher accuracy while significantly reducing memory and communication overheads.
| Subjects: | Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.22906 [cs.CL] |
| (or arXiv:2610.22906v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.22906 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Massimo Piccardi [view email]
[v1]
Fri, 16 Oct 2026 04:14:35 UTC (18,860 KB)
来源：arXiv cs.CL · arxiv.org