arXiv cs.CL· Ruurd J. A. Kuiper, Robert A. Bagheri·· 11 小时前AI 评分52
BYOD：仅用 LoRA 将自回归语言模型转换为扩散语言模型
BYOD: Build Your Own DLM - Efficient Autoregressive-to-Diffusion Language Model Conversion
AI 导读
研究团队提出 BYOD，一种仅用 LoRA 的参数高效方法，可将自回归语言模型转换为扩散语言模型，无需架构特定调优，已应用于 Gemma 2 9B、Llama 3.1 8B、Qwen2.5 7B 和 Ministral 8B，作者称这是首次报道的 Mistral 模型 AR-to-DLM 转换。
正文 · 原文
Abstract:Unlike autoregressive (AR) models, diffusion language models (DLMs) permit parallel token updates, flexible generation, and bidirectional attention, but leading systems rely on costly pretraining or full-model fine-tuning. We present Build Your Own DLM (BYOD), a generalizable LoRA-only recipe for parameter-efficient AR-to-DLM conversion. Without architecture-specific tuning, BYOD induces diffusion generation in Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B, and Ministral 8B; to our knowledge, this is the first reported AR-to-DLM conversion of a Mistral model. The adapters comprise 4.1-5.8% of final parameters while every base-model parameter remains frozen; each 25,000-iteration conversion needs at most 102.4M tokens, processed in 3.1-4.0 hours on one Google Colab GPU. All four models generate from fully masked sequences, use bidirectional context in a controlled attention test, and retain substantial parent-model knowledge. BYOD-Gemma is the strongest BYOD conversion on our nine-task evaluation, reaching 50.6% average accuracy compared with 51.9% for LLaDA 8B Instruct in the same harness. Open-generation quality at low network function evaluation (NFE) budgets remains the principal limitation: BYOD perplexity is 8.3-11.1 versus 4.1 for LLaDA and 2.2-3.0 for the AR parents when generating 128 tokens in 64 steps. To enable further experimentation and inspection, we provide a standalone training and evaluation implementation and interactive demos for all four conversions. BYOD thus provides a practical route for extending diffusion generation to other existing model families while identifying the gap that efficient conversion must next close.
| Comments: | 21 pages, 8 figures. Code: this https URL ; Gemma demo: this https URL . Links to all four demos are included in the paper |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.41574 [cs.CL] |
| (or arXiv:2610.41574v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.41574 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ruurd Jan Anthonius Kuiper [view email]
[v1]
Fri, 30 Oct 2026 11:04:29 UTC (419 KB)
来源：arXiv cs.CL · arxiv.org