arXiv cs.CL· Shangrui Nie, Florian Mai, David Kacz\'er, Charles Welch, Zhixue Zhao, Lucie Flek·· 13 小时前AI 评分31
从价值条件化到行为转变：LLM 的轻量级价值对齐
From Value Conditioning to Behavioral Shift: Lightweight Value Alignment of LLMs
AI 导读
研究提出用价值问卷式监督对开源 LLM 做轻量微调，无需大规模数据集或成对偏好标注即可改变模型的价值条件化输出。实验显示该方法能改变留出问卷问题的回答，并在 Reddit AITA 道德判断基准和文本交互决策基准等域外行为代理上产生响应分布变化，但迁移效果强烈依赖模型与具体价值，并受监督数据中价值描述框架方式影响。
正文 · 原文
Abstract:Large language models exhibit value-related preferences in their outputs, yet steering such behavior often requires large training datasets or paired preference annotations. In this work, we investigate a lightweight alternative: can fine-tuning on value survey responses induce measurable shifts in value-conditioned model outputs and downstream behavior? We first estimate baseline value profiles for several open-source LLMs by asking them to rate value-related descriptions organized around the ten Schwartz values. We then fine-tune models on survey-style supervision that selectively changes the target value ratings while keeping non-target values close to their baseline responses. We evaluate the resulting shifts in two settings. First, we measure whether the intervention changes responses on held-out survey questions. Second, we test whether these survey-induced shifts transfer to out-of-domain behavioral proxies, including a contextual moral-judgment benchmark constructed from Reddit AITA posts and a text-based interactive decision-making benchmark. Our results show that survey-based supervision can change held-out survey responses and produce downstream shifts in model response distributions on proxy behavioral evaluations. However, this transfer is strongly model- and value-dependent, and is shaped by how value descriptions are framed in the supervision data. These findings suggest that value surveys can serve not only as diagnostic tools, but also as lightweight supervision for studying behavioral value conditioning in LLMs.
| Comments: | 10 pages 2 figures |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2508.11414 [cs.CL] |
| (or arXiv:2508.11414v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2508.11414 arXiv-issued DOI via DataCite |
Submission history
From: Shangrui Nie [view email]
[v1]
Fri, 15 Aug 2025 11:36:17 UTC (926 KB)
[v2]
Wed, 21 Oct 2026 09:33:12 UTC (192 KB)
来源：arXiv cs.CL · arxiv.org