The research organization METR conducted a study to determine how modern AI tools affect the work of experienced developers. The study involved 16 developers who completed 246 real tasks in large projects they regularly work on. For half of the tasks, they were allowed to use tools like Cursor Pro and models Claude 3.5 and Claude 3.7 Sonnet, while for the others, AI usage was prohibited.
Before performing the tasks, participants expected AI to help reduce work time by 24 percent. However, the results showed the opposite: with AI, developers completed tasks on average 19 percent slower. Researchers note that some participants had limited experience specifically with Cursor, although most had already used web interfaces of large language models in their work.
When working with modern AI tools, developers spent more time formulating queries, waiting for responses, and checking code, rather than directly programming. This is especially noticeable in large and complex projects, where AI may struggle with context or require additional oversight.
METR emphasizes that they do not make definitive conclusions about AI’s effectiveness for all developers. Other studies have shown different results, and the tools themselves are rapidly evolving. However, this study demonstrates that the expected benefits of AI do not always match real-world experience, especially in complex open-source projects.
