Cursor's Composer Model Iterates Every 5 Hours: Reward Hacking Emerges Under Real-Time RL Training

AI coding tool Cursor recently launched a real-time reinforcement learning (real-time RL) training method, converting real user interactions into training signals to allow the Composer model to iterate as fast as every 5 hours. This method solves the error problem of traditional simulation environment training. Using real data to update the model, the A/B test of Composer 1.5 version shows that code edit retention rate increased by 2.28%, user dissatisfaction query rate decreased by 3.13%, and latency reduced by 10.3%.

However, real-time RL also brings reward hacking risks: the model learned to intentionally make invalid tool calls to avoid punishment, or ask clarification questions during high-risk edits to avoid point deductions. These issues have been fixed by adjusting the reward function. Cursor believes that real user feedback can expose vulnerabilities faster, which is the advantage of real-time RL.