Spaces:
Running
Running
| title: README | |
| emoji: 😻 | |
| colorFrom: purple | |
| colorTo: purple | |
| sdk: static | |
| pinned: false | |
| <p align="center"> | |
| <img src="https://huggingface.co/datasets/Program-Lab/ProgramLab/resolve/main/assets/logo.png" alt="ProgramLab" height="56"> | |
| </p> | |
| <p align="center"><b>Train single and multi agents to rebuild programs with RL</b></p> | |
| <p align="center"> | |
| <a href="https://programlab.blog/"><img alt="Blog" src="https://img.shields.io/badge/%F0%9F%93%9D%20Blog-programlab.blog-111111?style=for-the-badge"></a> | |
| <a href="https://github.com/programlab-agent/ProgramLab"><img alt="Code" src="https://img.shields.io/badge/Code-GitHub-181717?style=for-the-badge&logo=github"></a> | |
| <a href="https://huggingface.co/datasets/Program-Lab/ProgramLab"><img alt="Dataset" src="https://img.shields.io/badge/Dataset-ProgramLab-ffcc4d?style=for-the-badge&logo=huggingface"></a> | |
| </p> | |
| 1,261 RL environments where an agent runs a compiled program it cannot read, then rebuilds the whole program in any language. | |
| **Built for RL** | |
| - **Graded reward**: the reward is the pass rate over hidden behavioral tests. There are 187,070 in total, a median of 127 per task, and over 90% of tasks have at least 50, so partial progress earns partial reward. | |
| - **Validated reward**: tests that the original program fails, that an empty program passes, or that are flaky are removed. | |
| - **Hard to hack**: the agent never sees the source or the tests, and the reference program is execute-only and offline. | |
| - **Ready to run**: one prebuilt image per task, scored by the official ProgramBench evaluator. | |
| RL on ProgramLab raises Qwen3.6-35B-A3B on ProgramBench from 13.7% to 24.0% (Claude Code, two-hour budget). | |
| - [ProgramLab](https://huggingface.co/datasets/Program-Lab/ProgramLab): tasks, metadata, and build files | |
| - [ProgramLab-Tests](https://huggingface.co/datasets/Program-Lab/ProgramLab-Tests): hidden tests | |
| - Prebuilt task images: `ghcr.io/programlab-agent/programlab` | |