Coding Tool Olympics winner cost only 8¢ to reach the finish 🥇

Summary of AI Coding Tool Comparison

Key Concepts: AI coding assistants, code generation, prompt engineering, user interaction, cost analysis, V Code (Visual Studio Code with Copilot), Cursor, Windsurf, Claude with Desktop Commander, OpenAI models (DALL-E 2, DALL-E 3).

1. Experiment Setup:

• The experiment involves testing seven AI coding tools using the same prompt and model to isolate the impact of their internal prompting and tool usage on performance.
• The prompt given to the tools was: "Create for me an application that would allow me to compare evolution of image generation with AI from OpenAI models side by side. Show me DALL-E 2 from 2022, DALL-E 3 from 2023, and new image one."

2. V Code (Visual Studio Code with Copilot):

• Initial expectations were low due to past negative experiences.
• Performance was surprisingly better than expected, feeling similar to Cursor and Windsurf.
• The primary drawback was the high number of user confirmations required, making it the worst performer in terms of interruptions.
• Speed and cost were considered "okayish," placing it around fourth or fifth place.

3. Cursor and Windsurf:

• The user had prior experience with Cursor and then switched to Windsurf.
• Expected them to require more user interactions due to perceived friction.
• Surprisingly, they needed fewer interactions than anticipated.
• Cursor took gold with only eight interactions.
• Windsurf took silver with nine interactions.

4. Claude with Desktop Commander:

• Took bronze in terms of user interactions, requiring 13 interactions.

5. Cost Analysis:

• Claude Code achieved the lowest cost, completing the task in six messages at a cost of 1 cent per message (6 cents total with a Max subscription). Using API keys would cost 60 cents.
• Claude Desktop with Desktop Commander (Pro license) was second, requiring eight messages at 1 cent each (8 cents total).
• Windsurf took bronze with a cost of 21 cents.

6. User Interaction as a Metric:

• The number of user interactions needed to complete the task was a key performance indicator.
• Cursor performed best in this metric, followed by Windsurf and Claude Desktop.
• V Code required the most user interactions.

7. Notable Quotes:

• Regarding V Code with Copilot: "But surprisingly it actually was better than I expected. It felt very close to Corser and Wind and Windsor."
• Regarding Cursor and Windsurf: "surprisingly they needed less interactions."

8. Logical Connections:

• The experiment aims to isolate the performance differences between AI coding tools by controlling the prompt and model.
• The analysis covers both qualitative aspects (user experience, perceived friction) and quantitative metrics (number of interactions, cost).
• The comparison highlights the trade-offs between different tools in terms of user interaction, speed, and cost.

9. Synthesis/Conclusion:

The experiment reveals significant differences in the performance of AI coding tools, even when using the same prompt and model. Cursor and Windsurf excelled in minimizing user interactions, while Claude Code offered the lowest cost with a Max subscription. V Code with Copilot showed improvement but still lagged in user interaction. The choice of tool depends on the user's priorities, whether it's minimizing interruptions, reducing cost, or achieving a balance between the two.