SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations
Abstract
A benchmark of realistic software engineering assignments reveals that current coding agents struggle to track requirements from clarification through implementation, verification, and handoff, completing few tasks despite partial test success.
A coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains. Each assignment places an agent in a simulated engineering organization with 26 colleagues who have distinct responsibilities, bounded knowledge, and controlled collaboration behavior. Starting from a brief handoff, agents must recover requirements distributed across colleagues and workplace records, reconcile conflicting evidence, and implement changes evaluated by private executable tests. We evaluate eleven models and find a substantial gap between partial implementation and complete task delivery. The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but complete fewer than 8% of assignments. Providing complete requirements upfront improves F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra, yet leaves complete task success unchanged. Requirement-level trajectory analysis reveals failures throughout the delivery process: Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their accountable owners across runs but obtains only 64% of colleague-held requirements. Deferred questions go unfollowed, implementations remain incomplete even after requirements are acquired, and 31% of requirements reported as completed fail their tests. These findings highlight sustained requirement tracking, from clarification through implementation, verification, and handoff, as a key challenge for reliable autonomous software engineering.
Community
A coding agent working within a software organization must establish what to build, collaborate with
colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101
development assignments across 101 real repositories, 11 programming languages, and 12 application
domains. Each assignment places an agent in a simulated engineering organization with 26 colleagues
who have distinct responsibilities, bounded knowledge, and controlled collaboration behavior. Starting
from a brief handoff, agents must recover requirements distributed across colleagues and workplace
records, reconcile conflicting evidence, and implement changes evaluated by private executable tests.
We evaluate eleven models and find a substantial gap between partial implementation and complete
task delivery. The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but
complete fewer than 8% of assignments. Providing complete requirements upfront improves F2P by
approximately 16 percentage points for Seed Evolving and GPT-6 Astra, yet leaves complete task
success unchanged. Requirement-level trajectory analysis reveals failures throughout the delivery
process: Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their
accountable owners across runs but obtains only 64% of colleague-held requirements. Deferred
questions go unfollowed, implementations remain incomplete even after requirements are acquired,
and 31% of requirements reported as completed fail their tests.
Hi authors, thanks for the work. Two count questions: Figure 7(c) assigns 16 + 415 + 548 = 979 failed F2P nodes, but Figure 7(b) and Table 2 (63.84% of 2,699) leave at most 976 failed; and six Table 2 values do not fit the stated 2,699 F2P / 562 P2P totals (three P2P values fit 561). Details in this GitHub issue: https://github.com/TokenWave-AI/swe-cowork-bench/issues/1
Leading models passing about 63% of the tests but finishing under 8% of assignments matches how I'd feel handing a real handoff to an agent. I find it telling that 31% of requirements reported as done still fail. Did the agents that asked fewer questions upfront also report more false completions?
Get this paper in your agent:
hf papers read 2611.20218 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper
