arXiv cs.AI· Daniel Lenton, Yusha Arif, Haris Mahmood·· 4 小时前AI 评分55
Continual-ARC:衡量工作生命周期内的技能获取
Continual-ARC: Measuring Skill Acquisition over a Working Lifetime
AI 导读
研究者提出 Continual-ARC,用三版 ARC-AGI 构建终身任务流,任务在休眠间隔后以新实例重现,并按反馈预算消耗比例评分,使记忆型与权重型系统在同一尺度上比较。
正文 · 原文
Abstract:Agent harnesses have emerged as a promising mechanism for continual learning, alongside test-time parametric learning. However, the current benchmarks have several limitations: they generally focus on a single applied domain, serve each instance only once, and assume a single family of learner, so memory-based and weight-based systems are rarely compared on the same scale. We propose Continual-ARC, a lifelong stream built from the three ARC-AGI editions, whose tasks need no domain knowledge. The same tasks return after dormant gaps with fresh instances, while new tasks keep arriving. On each instance, the learner may pay for demonstrations, submit answers, or act inside the harness, with retries permitted on incorrect submissions, which are paid for. Each instance is scored based on the share of this feedback budget consumed. Since a learner may keep anything between instances, memory-based and weight-based systems are scored on the same scale. After the stream, the learner is frozen and tested on fresh instances of the tasks it met, which measures memorization, and on unseen tasks, which measures generalization. We run an agent harness with frozen weights, with and without persistence, on Qwen3.5-9B and GLM-5.3 Flash, comparing them with two prompting floors, a program-induction method and Qwen3.5-9B's weights trained to imitate GLM-5.3 Flash. Persistence in the harness generally does not improve memorization unless provided with explicit expert guidance. Most harness-based methods still learn less than a floor that simply caches each task's demonstrations, across all models we tested. Harness-based methods gain little or nothing on unseen tasks, whereas weight-based learning shows substantial gains. This opens the door to future weight-based continual learning that could teach models to drive harnesses in new domains.
| Comments: | 54 pages, 14 figures. Code and run records: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2611.14505 [cs.LG] |
| (or arXiv:2611.14505v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2611.14505 arXiv-issued DOI via DataCite |
Submission history
From: Daniel Lenton [view email]
[v1]
Tue, 10 Nov 2026 18:55:01 UTC (1,095 KB)
[v2]
Wed, 11 Nov 2026 05:12:57 UTC (1,095 KB)
[v3]
Sun, 15 Nov 2026 00:35:07 UTC (1,132 KB)
来源:arXiv cs.AI · arxiv.org