arXiv cs.AI· Eleftherios Triantafyllidis, Filippos Christianos, Zhibin Li, Bernd Bickel·· 23 小时前AI 评分31
WATERFALL:面向 POMDP 环境的自适应训练工作流,用进化式奖励生成与自动学习循环发现奖励
WATERFALL: Workflow for Adaptive Training with Evolutionary Reward Formulation and Automated Learning Loops
AI 导读
WATERFALL 是一套基于种群迭代的工作流,可在 POMDP 严格可观测性约束下自动发现奖励程序,无需目标条件、任务描述或专家示范。它通过 persona 条件生成多样化的程序化奖励候选,用视觉语言模型裁判的瑞士制进化锦标赛从原始视觉 rollout 中筛选精英,并依据纵向评估历史迭代变异、升级或剪枝候选。
正文 · 原文
Abstract:Reward design remains a central bottleneck in reinforcement learning (RL), particularly in sparse-reward, long-horizon and partially observable settings requiring dependency chaining, memory, and deceptive affordance disambiguation. While recent foundation-model approaches reduce manual reward engineering, most assume goal-conditioning, task descriptions, expert demonstrations, or ground-truth metrics. These assumptions are problematic in partially observable environments, where task-relevant objectives, affordances, and dynamics must be discovered through interaction rather than disclosed a priori. We formalise this problem setting by introducing WATERFALL, an iterative population-based workflow for automated reward discovery under the strict observability constraints inherent in POMDP environments. WATERFALL achieves this by (i) synthesising diverse programmatic reward candidates via persona-conditioned generation, (ii) evaluating candidate behaviour from raw visual rollouts utilising a Swiss-system evolutionary tournament judged by vision-language models to identify elites, and (iii) leveraging longitudinal assessment histories to iteratively mutate, escalate, or prune candidates. We evaluate WATERFALL across fully observable and reveal-gated partially observable environments ranging from discrete to continuous, comparing against sparse-reward RL, intrinsic-motivation and foundation-model reward-synthesis baselines. Our results indicate that WATERFALL discovers and refines reward programs without access to privileged information, outperforming baselines that retain their native information advantages, most clearly in the POMDP regime. Extensive ablations substantiate that visual evidence, tournament-based selection, and iterative refinement are jointly important. Enforcing this contract is what separates autonomous reward discovery from benchmark-privileged reward shaping.
| Comments: | Accepted at NeurIPS 2026 (poster). 78 pages, 13 figures, 19 tables. Code: this https URL OpenReview: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.22854 [cs.LG] |
| (or arXiv:2611.22854v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.22854 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Eleftherios Triantafyllidis [view email]
[v1]
Tue, 17 Nov 2026 14:35:27 UTC (19,706 KB)
来源:arXiv cs.AI · arxiv.org