Program Lab

Train single and multi agents to rebuild programs with RL

1,261 RL environments where an agent runs a compiled program it cannot read, then rebuilds the whole program in any language.

Built for RL

• Graded reward: the reward is the pass rate over hidden behavioral tests. There are 187,070 in total, a median of 127 per task, and over 90% of tasks have at least 50, so partial progress earns partial reward.
• Validated reward: tests that the original program fails, that an empty program passes, or that are flaky are removed.
• Hard to hack: the agent never sees the source or the tests, and the reference program is execute-only and offline.
• Ready to run: one prebuilt image per task, scored by the official ProgramBench evaluator.

RL on ProgramLab raises Qwen3.6-35B-A3B on ProgramBench from 13.7% to 24.0% (Claude Code, two-hour budget).

• ProgramLab: tasks, metadata, and build files
• ProgramLab-Tests: hidden tests
• Prebuilt task images: ghcr.io/programlab-agent/programlab