arXiv cs.CL· Tianjing Li, Wei Zhu·· 9 天前AI 评分22
Diffu-LoRA：面向个性化扩散模型的新型低秩适配方法
Diffu-LoRA: A Novel Low-Rank Adaptation for Personalized Diffusion Models
AI 导读
Diffu-LoRA 是一种参数高效的低秩适配方法，通过在 Transformer 块的线性层插入可训练低秩组件并为每个组件分配可学习门控，实现跨层非均匀的适配容量分配。
正文 · 原文
Abstract:Personalizing text-to-image diffusion models from a few reference images requires preserving subject identity while following prompts that describe new contexts. Full-model fine-tuning is parameter-intensive, whereas low-rank adaptation (LoRA) reduces the number of trainable parameters but leaves open how adaptation capacity should be distributed across layers. We introduce Diffu-LoRA, a parameter-efficient method that learns this allocation through gated low-rank adaptation. Diffu-LoRA inserts trainable low-rank components into the linear layers of Transformer blocks and assigns a learnable gate to each component. Bilevel optimization updates the adaptation weights and gate parameters on separate data splits, while progressive pruning removes components with the lowest gate values to meet a prescribed rank budget. This procedure allocates adaptation capacity nonuniformly across layers while keeping the pretrained backbone frozen. Experiments with Stable Diffusion on subjects from DreamBooth and additional collected datasets show improved overall subject fidelity and prompt alignment relative to the evaluated fine-tuning baselines. Ablation studies examine the contributions of bilevel optimization, progressive pruning, and adapter placement. These results support learned rank allocation as a practical approach to parameter-efficient diffusion model personalization.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.10550 [cs.CL] |
| (or arXiv:2610.10550v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10550 arXiv-issued DOI via DataCite |
Submission history
From: Wei Zhu [view email]
[v1]
Thu, 24 Sep 2026 16:57:09 UTC (253 KB)
来源：arXiv cs.CL · arxiv.org