arXiv cs.CL· Shanchao Liang, Nan Jiang, Shangshu Qian, Lin Tan·· 12 小时前AI 评分31
Waffle：用结构感知注意力与对比微调提升多模态模型 UI 转 HTML 能力
WAFFLE: Finetuning Multi-Modal Models for Automated Front-End Development
AI 导读
Waffle 是一种新的微调策略，通过结构感知注意力机制增强 LLM 对 HTML 层级结构的理解，并用对比微调对齐 UI 图像与 HTML 代码。
正文 · 原文
Abstract:Web development involves turning UI designs into functional webpages, which can be difficult for both beginners and experienced developers due to the complexity of HTML's hierarchical structures and styles. While Large Language Models (LLMs) have shown promise in generating source code, two major challenges persist in UI-to-HTML code generation: (1) effectively representing HTML's hierarchical structure for LLMs, and (2) bridging the gap between the visual nature of UI designs and the text-based format of HTML code. To tackle these challenges, we introduce Waffle, a new fine-tuning strategy that uses a structure-aware attention mechanism to improve LLMs' understanding of HTML's structure and a contrastive fine-tuning approach to align LLMs' understanding of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp (percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP, and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing benchmark Design2Code, outperforming current fine-tuning methods.
| Subjects: | Software Engineering (cs.SE); Computation and Language (cs.CL); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2410.18362 [cs.SE] |
| (or arXiv:2410.18362v4 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2410.18362 arXiv-issued DOI via DataCite |
Submission history
From: Shanchao Liang [view email]
[v1]
Thu, 24 Oct 2024 01:49:49 UTC (5,906 KB)
[v2]
Tue, 24 Jun 2025 20:35:02 UTC (5,316 KB)
[v3]
Mon, 2 Mar 2026 21:03:26 UTC (5,311 KB)
[v4]
Tue, 20 Oct 2026 04:48:34 UTC (5,313 KB)
来源：arXiv cs.CL · arxiv.org