Papers
arxiv:2611.20218

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

Published on Nov 15
· Submitted by
Jiaheng Liu
on Nov 17
Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,

Abstract

A benchmark of realistic software engineering assignments reveals that current coding agents struggle to track requirements from clarification through implementation, verification, and handoff, completing few tasks despite partial test success.

A coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains. Each assignment places an agent in a simulated engineering organization with 26 colleagues who have distinct responsibilities, bounded knowledge, and controlled collaboration behavior. Starting from a brief handoff, agents must recover requirements distributed across colleagues and workplace records, reconcile conflicting evidence, and implement changes evaluated by private executable tests. We evaluate eleven models and find a substantial gap between partial implementation and complete task delivery. The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but complete fewer than 8% of assignments. Providing complete requirements upfront improves F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra, yet leaves complete task success unchanged. Requirement-level trajectory analysis reveals failures throughout the delivery process: Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their accountable owners across runs but obtains only 64% of colleague-held requirements. Deferred questions go unfollowed, implementations remain incomplete even after requirements are acquired, and 31% of requirements reported as completed fail their tests. These findings highlight sustained requirement tracking, from clarification through implementation, verification, and handoff, as a key challenge for reliable autonomous software engineering.

Community

Paper submitter

A coding agent working within a software organization must establish what to build, collaborate with
colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101
development assignments across 101 real repositories, 11 programming languages, and 12 application
domains. Each assignment places an agent in a simulated engineering organization with 26 colleagues
who have distinct responsibilities, bounded knowledge, and controlled collaboration behavior. Starting
from a brief handoff, agents must recover requirements distributed across colleagues and workplace
records, reconcile conflicting evidence, and implement changes evaluated by private executable tests.
We evaluate eleven models and find a substantial gap between partial implementation and complete
task delivery. The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but
complete fewer than 8% of assignments. Providing complete requirements upfront improves F2P by
approximately 16 percentage points for Seed Evolving and GPT-6 Astra, yet leaves complete task
success unchanged. Requirement-level trajectory analysis reveals failures throughout the delivery
process: Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their
accountable owners across runs but obtains only 64% of colleague-held requirements. Deferred
questions go unfollowed, implementations remain incomplete even after requirements are acquired,
and 31% of requirements reported as completed fail their tests.

image

Hi authors, thanks for the work. Two count questions: Figure 7(c) assigns 16 + 415 + 548 = 979 failed F2P nodes, but Figure 7(b) and Table 2 (63.84% of 2,699) leave at most 976 failed; and six Table 2 values do not fit the stated 2,699 F2P / 562 P2P totals (three P2P values fit 561). Details in this GitHub issue: https://github.com/TokenWave-AI/swe-cowork-bench/issues/1

Leading models passing about 63% of the tests but finishing under 8% of assignments matches how I'd feel handing a real handoff to an agent. I find it telling that 31% of requirements reported as done still fail. Did the agents that asked fewer questions upfront also report more false completions?

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.20218
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.20218 in a model README.md to link it from this page.

Datasets citing this paper 1

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.20218 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.