arXiv cs.CL· Linhao Luo, Thuy-Trang Vu, Van-Anh Nguyen, Junae Kim, Gholamreza Haffari, Dinh Phung·· 11 小时前AI 评分31
MATO：面向大语言模型的多目标个性化对齐与测试时优化
MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models
AI 导读
MATO 是一个无需训练的框架，通过测试时优化在解码阶段以可控权重调节多个目标的相对重要性，无需修改模型参数或依赖外部奖励模型。其奖励发现模块从主干 LLM 构建提示词衍生的代理奖励，权重优化模块则根据用户初始偏好和部分生成结果动态调整权重。在多个数据集和主干 LLM 上的实验显示，MATO 持续优于强基线，实现帕累托改进的多目标对齐与更强的可控性。
正文 · 原文
Abstract:Aligning large language models (LLMs) with diverse and multifaceted user preferences is a fundamental challenge in personalized AI systems. Existing multi-objective alignment methods either rely on costly training or require pre-trained reward models for each preference, making it difficult for them to adapt to evolving preferences. Prompt-based personalization offers a training-free alternative, but prompting alone often provides limited steerability, as LLMs may overemphasize or overlook certain preferences and fail to give users reliable control over the relative importance of different objectives when conflicts arise, leading to suboptimal alignment. In this paper, we introduce MATO, a training-free framework for Multi-objective personalized Alignment with Test-time Optimization. MATO formulates personalization as a test-time optimization problem that steers the relative importance of multiple objectives through controllable weights during decoding, without modifying model parameters or requiring external reward models. Specifically, a reward discovery module constructs prompt-derived surrogate rewards from the backbone LLM for diverse objectives specified in natural language, while a weight optimization module dynamically adjusts objective weights based on the user's initial preferences and the partially generated response to balance competing objectives during generation. The resulting surrogate rewards and weights jointly guide an online optimization procedure over the token distribution, enabling better alignment with the target objectives. Extensive experiments across multiple datasets and backbone LLMs show that MATO consistently outperforms strong baselines, achieving Pareto-improving multi-objective alignment and stronger steerability. These results highlight test-time optimization as a promising direction for scalable, controllable, and model-agnostic personalized alignment.
| Comments: | Accepted by NeurIPS 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.25342 [cs.CL] |
| (or arXiv:2605.25342v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.25342 arXiv-issued DOI via DataCite |
Submission history
From: Linhao Luo [view email]
[v1]
Mon, 25 May 2026 01:57:22 UTC (7,691 KB)
[v2]
Fri, 6 Nov 2026 03:13:12 UTC (7,700 KB)
来源：arXiv cs.CL · arxiv.org