arXiv cs.CL· Jinlong Liu, Mohammed Bahja, Mark Lee·· 3 天前AI 评分27
推理监督反而有害：基于 TTCW 的长篇文学评论生成研究
When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation
AI 导读
研究构建了含 263,911 篇长篇故事、覆盖 14 个 TTCW 维度的数据集，测试推理轨迹能否提升结构化文学评论生成。在 Qwen3-4B、Qwen3-8B 和 Nemotron-8B 上，无轨迹监督的分数模仿保真度更高且报告生成近乎完整，而轨迹监督使完成率降至 85.9%–90.9%。失败分析显示 88.6% 的失败推理生成在轨迹内出现语义重复循环。
正文 · 原文
Abstract:Automatic evaluation of long-form literary writing remains challenging, particularly for creativity-related dimensions such as originality and flexibility. We address the lack of large-scale supervision for structured Torrance Test of Creative Writing (TTCW)-based evaluation by constructing a dataset of 263{,}911 long-form stories, each paired with scalar scores and meta-synthesised review comments across 14 TTCW-based dimensions. Using this dataset, we study whether reviewer-style reasoning traces improve structured literary review generation across Qwen3-4B, Qwen3-8B, and Nemotron-8B. Across all three models, no-trace supervision yields higher score imitation fidelity and near-complete report generation, whereas trace supervision reduces completion to $85.9$--$90.9\%$. Failure analysis shows that $88.6\%$ of failed reasoning generations contain semantic repetition loops inside the trace. These loops concentrate in particular story--criterion interactions and progressively harden after onset, while comparable repetition is rare in the supervision data. Our results suggest that, for long-form tasks requiring strictly structured outputs, explicit reasoning traces can introduce generation instability that outweighs their overall benefit.
| Comments: | Submit to NAACL 2027 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.20364 [cs.CL] |
| (or arXiv:2605.20364v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.20364 arXiv-issued DOI via DataCite |
Submission history
From: Jinlong Liu [view email]
[v1]
Tue, 19 May 2026 18:16:58 UTC (222 KB)
[v2]
Wed, 14 Oct 2026 11:05:55 UTC (321 KB)
来源：arXiv cs.CL · arxiv.org