GPT-5.5 leads in hacking tests, DeepSeek V4 Pro is cheapest

Security researcher Kasra Rahjerdi tested several AI models by making a fake book review app with a built-in security flaw. The app had exposed Firebase keys, and models had to unpack the app and find the keys to get into the database. Each model got a $10 budget and 2 hours, with total costs hitting $1,500.

GPT-5.5 solved the task 7 out of 10 times, costing $9.46 per success. DeepSeek V4 Pro succeeded 3 times but only cost $0.62 per success, about 1/15 of GPT-5.5's cost. Claude Sonnet 4.6 and Claude Opus 4.8 each succeeded twice, but Opus often got stopped by safety rules.

Gemini 3.1 Pro Preview almost always refused to start, using only about 9,000 tokens, far less than others' 100,000+. The researcher said the cost difference matters for teams running many security tools. Other models tested include GLM 5.1, Qwen 3.7 Max, and more.