arXiv cs.CL· Ziru Chen, Dongdong Chen, Ruinan Jin, Yingbin Liang, Yujia Xie, Huan Sun·· 6 天前AI 评分25
COBALT：用离线轨迹的上下文赌博机学习实现多轮代码生成
COBALT: Contextual Bandit Learning with Offline Trajectories for Multi-Turn Code Generation
AI 导读
COBALT 提出一种复用离线多轮轨迹训练 LLM 做多轮代码生成的方法，基于多轮代码生成是一步可恢复马尔可夫决策过程的观察，先用参考 LLM 收集轨迹并切分为部分轨迹，再通过单步代码生成完成每个部分轨迹。
正文 · 原文
Abstract:Training large language models (LLMs) for multi-turn code generation typically relies on online reinforcement learning (RL), in which models learn from repeatedly generated online rollouts. Although effective, online RL can be costly and unstable. In this paper, we present contextual bandit learning with offline trajectories (COBALT), a simple method that reuses offline multi-turn trajectories to efficiently train LLMs for multi-turn code generation. Building on the observation that multi-turn code generation is a one-step recoverable Markov decision process, COBALT first collects code generation trajectories using a reference LLM and divides them into partial trajectories. Then, during online bandit learning, the LLM is trained to complete each partial trajectory through single-step code generation. On LiveCodeBench, COBALT substantially improves R1-Distill 8B and Qwen3 8B by up to 9.0 and 6.2 absolute Pass@1 points and matches the performance of two multi-turn online RL baselines using GRPO and VeRPO. We also perform a perturbation-based analysis of in-context reward hacking and show that Cobalt can improve robustness to misleading execution feedback. Overall, our results demonstrate that Cobalt is a promising solution for multi-turn code generation. Our code and data are available at this https URL.
| Comments: | TMLR camera ready |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Software Engineering (cs.SE) |
| Cite as: | arXiv:2602.03806 [cs.LG] |
| (or arXiv:2602.03806v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.03806 arXiv-issued DOI via DataCite |
Submission history
From: Ziru Chen [view email]
[v1]
Tue, 3 Feb 2026 18:08:41 UTC (265 KB)
[v2]
Thu, 8 Oct 2026 22:26:23 UTC (1,723 KB)
来源：arXiv cs.CL · arxiv.org