E-SPL：面向 LLM 强化学习的进化式系统提示词学习

View PDF HTML (experimental)

Abstract:Building agentic systems that can autonomously self-improve from experience is a longstanding goal of AI. Large language models (LLMs) today primarily self-improve via two mechanisms: self-reflection for context updates, and reinforcement learning (RL) for weight updates. In this work, we propose Evolutionary System Prompt Learning (E-SPL), a method for jointly improving a model's experiential context and its weights. In each RL iteration, E-SPL samples trajectories under multiple system prompts in parallel, then uses them to jointly update the weights with RL and evolve the system prompts with self-reflection. E-SPL encourages a natural division between procedural knowledge in weights and interpretable declarative knowledge in system prompts. Most notably, in an easy-to-hard generalization setting (AIME $\rightarrow$ BeyondAIME), E-SPL improves RL success rate from 38.8% $\rightarrow$ 45.1%. E-SPL also improves RL on AIME 2025 (56.3% $\rightarrow$ 60.6%), HMMT 2025 (50.0% $\rightarrow$ 52.7%), and agentic search (44.2% $\rightarrow$ 48.6% on gpt-oss-120b). Across all settings, E-SPL outperforms both RL-only and evolution-only baselines, demonstrating that weight updates and context updates are deeply synergistic and can together yield gains in generalization that neither achieves alone. Code is available at this https URL.

Submission history

From: Lunjun Zhang [view email]
[v1] Mon, 16 Feb 2026 12:34:27 UTC (4,259 KB)
[v2] Mon, 23 Feb 2026 08:43:06 UTC (4,246 KB)
[v3] Wed, 25 Feb 2026 03:50:21 UTC (4,248 KB)
[v4] Wed, 4 Nov 2026 08:38:33 UTC (6,576 KB)