跳到正文
arXiv cs.AI· Yuheng Chen, Fan Zhang, Yixuan Xu, Chao Yang, Yang Xiang, Can Yang·· 23 小时前AI 评分31

生成式推荐中 RL 后训练的隐藏代价:排序更强,召回更差

Better Rankers, Worse Retrievers: The Hidden Cost of RL Post-Training in Generative Recommendation

AI 导读

研究发现,生成式推荐模型经 RL 后训练后,在小 K 的 hit@K 上更准,但在全部五个数据集上候选列表的召回(K 达数百)反而下降,即“排序更强、召回更差”。

正文 · 原文

View PDF HTML (experimental)

Abstract:Generative recommenders decode the semantic ID of a user's next item with a language model, and post-training them with reinforcement learning (RL) after supervised fine-tuning (SFT) has become standard practice. The gains are mostly reported on the first ten results, yet in deployment such models hand hundreds of candidates to a downstream ranker or reward model. We show that RL post-training has a hidden cost: it improves accuracy at the top of the list (hit@K for small K) but, on all five datasets we study, lowers the recall of the candidate list (hit@K for K in the hundreds). The model becomes a better ranker and a worse retriever. Controlled experiments trace the loss to the negative advantages that RL assigns to the candidates it samples, the rollout. The root cause is that only one next item is logged per user, so RL treats every other candidate as wrong. Yet these candidates are the model's most plausible alternatives, and often the next items of other users with similar histories. Penalising them pushes down exactly what similar users choose next, and these items drop out of those users' candidate lists. This diagnosis points directly to a remedy: out-of-rollout anchoring, a single additional loss term that preserves the SFT model's ordering of the items outside the rollout. It keeps the gain at the top and restores recall to the SFT level on all five datasets, and it is effective under other RL configurations as well. To our knowledge, this is the first systematic study of this recall loss: we measure it, identify its cause, and show that, once the cause is known, a simple remedy suffices.
Subjects: Information Retrieval (cs.IR); Artificial Intelligence (cs.AI)
Cite as: arXiv:2611.22838 [cs.IR]
  (or arXiv:2611.22838v1 [cs.IR] for this version)
  https://doi.org/10.48550/arXiv.2611.22838

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yuheng Chen [view email]
[v1] Tue, 17 Nov 2026 14:19:22 UTC (1,194 KB)

来源:arXiv cs.AI · arxiv.org