Papers
arxiv:2611.14505

Continual-ARC: Measuring Skill Acquisition over a Working Lifetime

Published on Nov 11
· Submitted by
Daniel Lenton
on Nov 16
Authors:
,

Abstract

Continual-ARC, a lifelong stream of ARC-AGI tasks, compares memory- and weight-based continual learning under frozen-weight agent harnesses, revealing that harness persistence rarely improves memorization and most harness methods underperform simple caching, while weight-based learning shows better generalization.

Agent harnesses have emerged as a promising mechanism for continual learning, alongside test-time parametric learning. However, the current benchmarks have several limitations: they generally focus on a single applied domain, serve each instance only once, and assume a single family of learner, so memory-based and weight-based systems are rarely compared on the same scale. We propose Continual-ARC, a lifelong stream built from the three ARC-AGI editions, whose tasks need no domain knowledge. The same tasks return after dormant gaps with fresh instances, while new tasks keep arriving. On each instance, the learner may pay for demonstrations, submit answers, or act inside the harness, with retries permitted on incorrect submissions, which are paid for. Each instance is scored based on the share of this feedback budget consumed. Since a learner may keep anything between instances, memory-based and weight-based systems are scored on the same scale. After the stream, the learner is frozen and tested on fresh instances of the tasks it met, which measures memorization, and on unseen tasks, which measures generalization. We run an agent harness with frozen weights, with and without persistence, on Qwen3.5-9B and GLM-5.3 Flash, comparing them with two prompting floors, a program-induction method and Qwen3.5-9B's weights trained to imitate GLM-5.3 Flash. Persistence in the harness generally does not improve memorization unless provided with explicit expert guidance. Most harness-based methods still learn less than a floor that simply caches each task's demonstrations, across all models we tested. No harness-based method gains on unseen tasks, whereas weight-based learning shows substantial gains. This opens the door to future weight-based continual learning that could teach models to drive harnesses in new domains.

Community

Paper author Paper submitter

We built Continual-ARC to see whether agents get better with experience, scoring notes in a harness and updates to the weights on the same scale. ARC tasks recur through a stream of 200 instances, each time with a fresh input, and every demonstration or wrong answer costs the learner part of a fixed budget. After the stream we freeze the learner and test it on new instances of the tasks it met (memorization) and on 100 tasks it never met (generalization).

Out of the box, most harnesses didn't learn measurably, and most of the harness-based methods we tried learned less than a baseline that just caches each task's demonstrations. Told explicitly to save a verified program per task, all three harnesses we ran on GPT-6 Luna memorized, but they gained little or nothing on unseen tasks. Training the weights did generalize: Qwen3.5-9B, trained with DAgger to imitate GLM-5.3 Flash, spent 29% less than its untrained self on the 100 unseen tasks.

Caveat: it's one domain, and most of our runs use fairly small models. Would love to hear where you think the benchmark falls short!

Having the same tasks return with fresh instances is a nice way to separate memorizing from actually learning, and it is a little humbling that plain caching of demonstrations beat most harness methods. You asked where it might fall short: I wonder whether 100 unseen tasks is enough to tell real generalization apart from luck on a few easy ones.

·

Thanks! Each learner is scored against itself starting from scratch on the same 100 unseen tasks, and the 95% intervals come from resampling those tasks, so a saving carried by a few easy ones would get a wide interval. DAgger spent 29% less than its untrained self on the unseen tasks, with a 95% interval of 23% to 35%, and it solved 47 to 52 of the 100 per play where the untrained model solved 13 to 26. The intervals don't capture how much a whole training run varies, though, since they're conditioned on one trained model from one stream.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.14505
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.14505 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.14505 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.14505 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.