arXiv cs.CL· Arman Nik Khah, Arvin Bahreini·· 13 小时前AI 评分39
研究发现后训练让模型写作更可预测，但作为评审几乎不偏向可预测写作
Is the Judge Its Twin? Post-training makes a model's writing more predictable but barely moves its taste, as a judge, toward predictable writing
AI 导读
研究追踪 OLMo-2 和 Zephyr 两个 70 亿参数模型系列从基座到 SFT、DPO 各阶段，发现模型作为作者时写作可预测性按基座、SFT、DPO 顺序递增，另一模型系列判断其故事每 token 惊讶度低 7.0% 至 9.2%（OLMo-2）和 24% 至 25%（Zephyr）。
正文 · 原文
Abstract:Language models now routinely grade other language models. If post-training makes a model's writing more predictable, it may also teach the same model, as a judge, to reward predictable writing. We test this directly. We follow two open 7-billion-parameter model families, OLMo-2 and Zephyr, from their base models through supervised fine-tuning (SFT) and direct preference optimization (DPO), and measure every stage as a writer of short stories and as a judge of story pairs. As writers, the models drift as feared. Each family's trained model finds its stages' stories more familiar in the order base, SFT, DPO, in all ten prompts, and a model from the other family finds the trained stories 7.0 to 9.2 percent (OLMo-2) and 24 to 25 percent (Zephyr) less surprising per token. As judges, they barely move toward predictable writing. Asked which story is better, a question every judge passes a competence check on, no trained judge's preference for the more predictable story grows by as much as one point of pick probability at its estimate. A one-sided 95% upper bound on that growth is about 5 points on the widest pairs. Training instead makes judges more swayed by story length and by display order, and it breaks "more creative" as a question. Trained judges no longer reliably prefer a story to its own words in scrambled order. For these two families, the practical hazard in a small trained judge of creative writing lies in these measurable habits.
| Comments: | v2: title softened, editorial revisions throughout, two references added; no results changed. 16 pages, 4 tables. Code and data: this https URL |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| ACM classes: | I.2.7; I.2.6 |
| Cite as: | arXiv:2609.32196 [cs.CL] |
| (or arXiv:2609.32196v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.32196 arXiv-issued DOI via DataCite |
Submission history
From: Arman Nik Khah [view email]
[v1]
Sat, 26 Sep 2026 03:46:50 UTC (21 KB)
[v2]
Thu, 22 Oct 2026 13:40:32 UTC (21 KB)
来源：arXiv cs.CL · arxiv.org