arXiv cs.AI· Akansha Kalra, Ellen R. Novoseller, Vernon J. Lawhern, Daniel S. Brown·· 12 小时前AI 评分31
P3L：无需标注者身份的多元偏好对齐框架
Steerable Pluralistic Policy Alignment from Anonymized Crowd Feedback
AI 导读
研究者提出 Preference-PCA Pluralistic Learning（P3L），一种无需标注者身份、从匿名众包反馈中通过 PCA 分解联合学习轨迹嵌入与偏好方向共享基的多元对齐框架。在多个具身控制任务上，P3L 实现可引导的个性化对齐，可泛化到离线数据集外的新用户，并保持有竞争力的 RL 与测试表现，同时大幅降低计算成本、支持可变的适配查询预算。
正文 · 原文
Abstract:Pluralistic alignment requires AI systems to learn from diverse populations whose preferences may reflect heterogeneous or even conflicting objectives, and to personalize their behavior to individual users. Existing approaches in pluralistic alignment often assume access to annotator identities and rely on powerful pretrained representations to model individual preference variations. We propose Preference-PCA Pluralistic Learning (P3L), a novel framework for pluralistic alignment that jointly learns trajectory embeddings and the shared basis of preference directions via PCA decomposition from heterogeneous crowd feedback without requiring annotator identities. At test time, our approach enables rapid and accurate personalized reward learning and downstream policy optimization. Experiments across multiple embodied control tasks demonstrate that P3L enables steerable personal alignment that generalizes beyond the offline dataset to out-of-distribution new users and attains competitive RL and test performance, while substantially reducing computational cost and supporting variable adaptation query budgets.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO) |
| Cite as: | arXiv:2610.26914 [cs.LG] |
| (or arXiv:2610.26914v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.26914 arXiv-issued DOI via DataCite |
Submission history
From: Akansha Kalra [view email]
[v1]
Sat, 17 Oct 2026 06:14:59 UTC (2,609 KB)
来源：arXiv cs.AI · arxiv.org