SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios

作者：Minh-Tin Thai, Tue Le, Dũng Nguyễn Mạnh, Huy Phan Nhat, Nghi D. Q. Bui · 发表于：arXiv.org · 年份：2025 · DOI：10.48550/arxiv.2512.18470 · 被引用次数：40 · 研究领域：Computer Science

Existing benchmarks for AI coding agents focus on isolated, single-issue tasks such as fixing a bug or adding a small feature. However, real-world software engineering is a long-horizon endeavor: developers interpret high-level requirements, coordinate changes across many files, and evolve codebases over multiple iterations while preserving functionality. We introduce SWE-EVO, a benchmark for this long-horizon software evolution challenge. Constructed from release notes of seven mature open-source Python projects, SWE-EVO comprises 48 tasks requiring multi-step modifications spanning an average of 21 files, validated against test suites averaging 874 tests per instance. Experiments reveal a striking capability gap: GPT-5.4 with OpenHands achieves only 25% on SWE-EVO versus 72.80% achieved by GPT-5.2 on SWE-Bench Verified, showing that current agents struggle with sustained, multi-file reasoning. We also propose Fix Rate, a metric capturing partial progress on these complex, long-horizon tasks.