OpenAI says 30% of SWE-Bench Pro tasks have flaws

OpenAI posted a blog challenging SWE-Bench Pro, a coding benchmark for AI models. Created by Scale AI, it's known for being close to real-world development and hard to cheat. OpenAI found about 30% of 731 public tasks have flaws.

OpenAI says top models' pass rate jumped from 23.3% to 80.3% in 8 months, making the benchmark less useful. Their data analysis flagged 200 failed tasks, while human checks found 249. Issues include tests being too strict, incomplete hints, narrow scopes, and misleading prompts.

A typical case: the task asked for 1 space at line start for Markdown, but hidden tests required 2 spaces, causing models to fail. OpenAI has dropped using this benchmark and wants new ones designed by senior developers.