arXiv cs.CL· Siliang Zeng, Luca Viano, Chenliang Li, Jiaxiang Li, Volkan Cevher, Markus Wulfmeier, Stefano Ermon, Alfredo Garcia, Mingyi Hong·· 12 小时前AI 评分31
大语言模型与人类反馈对齐的数学基础与算法设计
Aligning Large Language Models with Human Feedback: Mathematical Foundations and Algorithm Design
AI 导读
一篇教程式文章系统梳理了将 LLM 与人类意图、偏好和价值观对齐的数学基础与算法框架，涵盖 SFT、RLHF 和 DPO 等标准对齐技术。文章还探讨了从人类偏好学习的理论根基，将其与逆强化学习（IRL）和离散选择模型建立联系，并讨论了各算法的优势、局限与实践实现要点。
正文 · 原文
Abstract:This article provides an introduction to the mathematical foundations and algorithmic frameworks used to align Large Language Models (LLMs) with human intentions, preferences, and values. We discuss standard alignment techniques, such as fine-tuning (SFT), reinforcement learning with human feedback (RLHF), and direct preference optimization (DPO). We also explore the theoretical underpinnings of learning from human preferences, drawing connections to inverse reinforcement learning (IRL) and discrete choice models. We present state-of-the-art algorithms in a tutorial style, discuss their advantages and limitations, and offer insights into practical implementation. Our exposition is intended to serve as a comprehensive resource for researchers and practitioners, providing both a foundational understanding of alignment methodologies and a framework for developing more robust and scalable alignment techniques.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.31862 [cs.LG] |
| (or arXiv:2610.31862v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.31862 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | 2026 Special Issue on the Mathematics of Deep Learning]. IEEE Signal Processing Magazine, 43(3), 71-85 |
| Related DOI: | https://doi.org/10.1109/MSP.2026.3666824
DOI(s) linking to related resources |
Submission history
From: Chenliang Li [view email]
[v1]
Thu, 22 Oct 2026 19:50:27 UTC (562 KB)
来源：arXiv cs.CL · arxiv.org