arXiv cs.CL· Ajit Mallavarapu, Ziwei Gu·· 11 小时前AI 评分29
无需训练也能发现 LLM 风格维度：采样加 PCA 即可识别提示词相关风格轴
Sampling Reveals Style: Unsupervised, Training-Free Discovery of Prompt-Conditional Stylistic Axes in LLM Activations
AI 导读
研究者提出一种无需训练、针对单条提示词的方法：对同一提示词在较高温度下反复采样，对池化的隐藏激活做 PCA，再用两极生成自动标注风格轴。在 Qwen-3.5-4B-Instruct 上，前两个轴与人类自发提出的风格维度匹配的精确率为 72.8%、宏召回率 43.6%，75.6% 的有效性评分认为两极生成与标签相符。
正文 · 原文
Abstract:Large language models (LLMs) encode rich stylistic structure in their hidden activations, but discovering which stylistic dimensions are salient for a given prompt typically requires supervised contrastive data. We present a training-free, prompt-conditional alternative: we repeatedly sample completions of a single prompt at elevated temperature, apply Principal Component Analysis (PCA) to the pooled hidden activations, and label the resulting axes automatically from the pole generations. We validate the discovered axes against 245 human-elicited stylistic annotations in a two-phase study. On our strongest model (Qwen-3.5-4B-Instruct), the top two axes match spontaneously requested human dimensions with 72.8% precision and 43.6% macro-recall, and 75.6% of validity ratings judge the axes' polar generations accurate to their labels, with 90.9% adjacent inter-annotator agreement. Discoverability is strongly model-dependent: both Qwen models and Llama-3.2-3B expose human-salient axes, while DeepSeek-7B-Chat drops to 35.3% precision, its leading components dominated by structural rather than stylistic variance. Simple PCA over a model's own decoding variance is thus an effective, low-cost probe of stylistic structure in LLM representations, one that also exposes sharp cross-model differences in how that structure is organized.
| Comments: | Published at the BlackboxNLP Workshop @ EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.19150 [cs.CL] |
| (or arXiv:2609.19150v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.19150 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the 9th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 639-654, 2026 |
| Related DOI: | https://doi.org/10.18653/2026.blackboxnlp-1.37
DOI(s) linking to related resources |
Submission history
From: Ziwei Gu [view email]
[v1]
Mon, 20 Jul 2026 11:55:26 UTC (480 KB)
[v2]
Fri, 18 Sep 2026 12:42:25 UTC (468 KB)
[v3]
Fri, 6 Nov 2026 00:03:05 UTC (469 KB)
来源：arXiv cs.CL · arxiv.org