SWE-bench

What SWE-bench is

SWE-bench is a benchmark for evaluating language models on software issues collected from GitHub. A model generates a patch for an issue; the evaluation harness applies it in Docker and runs the repository tests to check whether the issue is resolved. The original benchmark contains 2,294 tasks from 12 Python repositories. Other available datasets include Lite, Verified, Multimodal, and Multilingual: Verified has 500 engineer-checked solvable instances, while Multilingual covers 300 instances across 9 languages and 42 repositories. Multimodal has 100 development instances with screenshots and interface elements, with test evaluation through the SWE-bench API. Its CLI can submit predictions, retrieve reports, manage evaluation runs, and track cloud performance. Evaluations can run locally or through cloud options using Modal and sb-cli. The project is free. Installation requires Python 3.9 or later and Docker; setup instructions cover macOS, Linux, and Windows, with WSL 2 required for Windows. The Docker guide recommends at least 120 GB of free disk space and 16 GB of RAM; caching all Docker images can require about 2,000 GB.