arXiv cs.CL· Liwei Zhang, ZhiMing Tong·· 10 小时前AI 评分38
一个字节一个 Token：用自生成数据将 27B 混合注意力 LLM 字节化
One Byte, One Token: Byteifying a 27B Hybrid-Attention LLM with Self-Generated Data
AI 导读
研究者将 Qwen3.8-27B（48 层 Gated DeltaNet + 16 层全注意力）的词表从 248,320 缩减到 384 个条目，其余权重不动，用 4.77 GB 自生成监督字节做两阶段续训，在 8xH200 上耗时 178.6 h，转为纯字节级模型。
正文 · 原文
Abstract:Byte-level BPE fragments Chinese text (for Qwen3.8 on Chinese Wikipedia, 3.84 bytes per token on average; 64% of tokens are single characters) and hides spelling from the model. We ask whether an already-trained 27B subword model can be converted directly into a pure byte-level model (one byte, one token), without a patching structure or a local encoder, while keeping most of its ability. Starting from Qwen3.8-27B (48 Gated DeltaNet layers + 16 full-attention layers), we shrink the vocabulary from 248,320 to 384 entries (the 256 byte symbols already in the BPE vocabulary plus special ids), leave every other weight untouched, and continue training in a two-stage curriculum (encoding first, then reading) on 4.77 GB of supervised bytes: all dialogue/behavioral supervision is generated on-policy by the original model itself (character dictionary, word dictionary, general dialogue, coding sessions), and knowledge reading uses encyclopedia text (54% of supervised bytes). Training takes 178.6 h on 8xH200. On the same GPU and serving stack, the byte model's gap to the original falls within the paired bootstrap interval on MATH-500, HumanEval+, MMLU-Pro and non-thinking GSM8K, and it loses 4-6 points on C-Eval, thinking-mode GSM8K and non-thinking IFEval. Without thinking it leads on every character-level task (CUTE 75.2 -> 84.0, EXECUTE-zho 81.6 -> 93.9, our Chinese character suite 54.4 -> 64.0), but lookup knowledge about Chinese characters such as pronunciation and stroke count drops, and thinking does not recover it. The cost is throughput: per-step speed is identical, but each step yields 3-5x less output; Chinese characters per second is about 1/4 of the original, and prefill at 64K bytes is 6.5x slower. We also document four byte-model failure modes (greedy runaway, sampling breaking UTF-8, ineffective penalty terms on a small vocabulary, repetition) and their fixes.
| Comments: | Full text in English followed by the full text in Chinese. 39 pages, 2 figures, 7 tables. Code and task suite: this https URL |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2611.13607 [cs.CL] |
| (or arXiv:2611.13607v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2611.13607 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Liwei Zhang [view email]
[v1]
Tue, 10 Nov 2026 05:33:41 UTC (64 KB)
来源：arXiv cs.CL · arxiv.org