← back to the archiveCover illustration for “SWE-bench: the benchmark for AI fixing real GitHub issues”
TERM6mo ago·Published · Updated ·by Andy Padia

SWE-bench: the benchmark for AI fixing real GitHub issues

In short: SWE-bench tests whether AI systems can resolve real GitHub issues in open-source Python repositories, judged by the project's own tests.

SWE-bench is a benchmark that measures whether an AI system can resolve real GitHub issues from popular open-source Python repositories. Each task gives the issue text and the codebase; the system must produce a patch, which is judged by running the project's tests. The original paper, submitted on 10 October 2023, built 2,294 tasks from 12 repositories.

SWE-bench in plain words

Instead of quiz questions, the AI gets real bug reports from real projects, and the only thing that counts is whether the fix passes the tests.

How it works

Tasks come from merged pull requests that resolved an issue and contributed tests. The system receives the issue and the repository as it stood before the fix, and produces a patch. The harness then runs fail-to-pass tests, which the fix should make pass, alongside pass-to-pass tests that were already passing. The score is the percentage of tasks resolved.

When the paper came out, the best model it tested, Claude 2, resolved 1.96% of the issues. The SWE-bench site now hosts several leaderboards. One is SWE-bench Verified, a subset of 500 instances created with OpenAI, in which human annotators checked each task for a clear problem description and correct tests.

Example

As an illustration, a task might be an issue reporting that a date-parsing function mishandles a timezone format. The agent must find the function, change it and pass the hidden tests the original maintainers wrote for that fix.

Where you'll meet it

  • Model and coding-agent launch announcements
  • Leaderboards comparing agents and harnesses
  • Variants such as Verified, Lite, Multilingual and Multimodal
  • Research on agent design and evaluation

What it changes for you

For engineering leaders, a high score shows an agent can handle real, bounded maintenance tasks in these repositories. My reading is that well-known public Python projects with good tests flatter agents compared with a private, poorly tested codebase. Scores also reflect the harness as well as the model: the site keeps a separate bash-only leaderboard that runs models through one minimal agent, mini-SWE-agent.

SWE-bench vs HumanEval

HumanEval tests writing a standalone function from a docstring, across 164 hand-written problems. SWE-bench tests changing a real codebase to fix an issue, which is closer to actual engineering work.

Common mistake

My view: reading a SWE-bench score as "percentage of my team's tickets the agent can close". Your repositories, tests and tickets differ. Pick twenty closed tickets from your own backlog and replay them; that is your benchmark.

Agentic coding · Benchmark contamination · Pass@k · Agent harness · Evals

From the archive: Coding agents need a refinement budget

Replay twenty closed tickets from your own backlog, because that is the benchmark that describes your team.

#field-guide#evaluation#benchmarks#coding-agents

Sources

  1. arxiv.org/abs/2310.06770
  2. swebench.com
  3. swebench.com/verified.html
  4. arxiv.org/abs/2107.03374
← older drop
Spec-driven development: write the spec, let agents build
newer drop →
HNSW: the index behind fast vector search

related drops

explore all 557 drops →
← back to the archiveday 175