yes, AI makes developers slower (and how to prevent it)

Key Concepts

AI in programming, developer productivity, LLMs (Large Language Models), Cursor Pro, Claude 3.7, GitHub study, randomized control trial, AI reliance, MVP (Minimum Viable Product), context engineering, Claude Code, custom hooks, ultra think, AI agents, Vectoral, GitHub Copilot, agent orchestration, iterative collaboration, critical verification, prompt engineering.

Study on AI Impact on Developer Speed

Introduction

The video discusses a study by Meter that investigated the impact of AI tools on developer speed. Contrary to the common belief that AI speeds up programming, the study found that AI actually slowed down developers by 19%. The video aims to explain why this happened and how developers can avoid these pitfalls to boost productivity.

Study Details

• Participants: 16 senior open-source maintainers with an average of 10 years of experience. 93% had prior experience with LLMs, and 44% had used Cursor.
• AI Tool: Cursor Pro with Claude 3.7.
• Methodology: Randomized control trial with 246 GitHub issues. Developers flipped a coin to determine whether they would use AI for a given task.
• Measurement: Time taken from the start of work on an issue to the pull request being reviewed, approved, and merged.
• Prediction vs. Reality: Researchers predicted a 24% speedup with AI, but the study found a 19% slowdown.

Reasons for Slowdown

1. Lack of Server Reliance: Developers relied on AI even for simple tasks that they could have completed faster manually.
2. Existing Knowledge: The developers already possessed significant knowledge of the codebases, limiting the value AI could add.
3. Codebase Size: The large size of the codebases (over a million lines) overwhelmed the AI models.

Criticisms of the Study

• Small Sample Size: Only 16 developers participated.
• Senior Developer Bias: Senior developers have established workflows and may be more resistant to AI-generated code.
• Limited Toolset: The study only used Cursor and did not explore more advanced AI agents.
• Focus on Existing Repositories: The study did not assess AI's impact on building new projects from scratch.

GitHub Study and Shift in Ambition

The video references a GitHub study that found AI to be time-saving and highlighted a shift in developer ambition. The CEO of GitHub, Thomas Dome, suggests that AI is on track to write 90% of code within the next two to five years. The study emphasized the importance of new skills like agent orchestration, iterative collaboration, critical verification, and prompt engineering.

Seven Tips for Effective Coding with AI

1. Use the Right Model: Choose LLMs based on context size, benchmark scores, and personal testing. Verify code generated by one model with a different model from a different company (e.g., Claude-generated code with Gemini).
2. Build New Features: AI provides significant speedups when starting new projects or adding new features (10x-100x).
3. Small, Focused Changes on Large Codebases:
   • Make small, focused changes.
   • Avoid large refactors in one go; split them into stages.
   • Work in a feature branch.
   • Commit often (every 10-15 minutes).
4. Avoid Errors from the Start:
   • Have the AI analyze the codebase.
   • Provide the LLM with ample context.
   • Understand the feature or bug yourself first.
   • Run linters, type checks, and automated tests.
5. Discipline Equals Speed: Spend more time planning and understanding the task before having the AI write code. Create a markdown file outlining the vision, steps, and unknowns.
6. Context Engineering:
   • Create clear prompts with specific inputs, outputs, and constraints.
   • Use a markdown file as a scratchpad for the AI agent.
   • Tag relevant files.
   • Build internal documentation for the codebase.
7. Human as Decision Maker:
   • Maintain the big picture view.
   • Don't blindly trust the AI.
   • Use AI for its strengths, but rely on human judgment for creativity, taste, and architecture.
   • Use AI to understand the root cause of issues.
   • Prioritize upskilling yourself with AI.
   • Listen to users and build what they want.

Claude Code Custom Hooks

The video introduces custom hooks for Claude Code, which are scripts that run before or after Claude performs an action.

• Append Ultra Think: Adds "Use the maximum amount of ultra think. Take all the time you need. It's much better if you do too much research and thinking than not enough." to prompts ending with "-u". This influences Claude's reasoning effort.
• Explain Logs: Adds "Above are the relevant logs. Your job is to think harder about what these logs say. Give me a simpler and short explanation. Do not jump to conclusions. Do not make assumptions. Quiet your ego. Assume you know nothing." to prompts ending with "-e". This helps Claude better understand and explain logs.
• Golden Prompt: Adds "Think harder answer in short. Keep it simple." to prompts ending with "-d". This improves the output quality.

The video explains how to add these hooks to the settings.json file in the Claude folder. It also mentions other possible hooks, such as pre-tool use hook and post-tool use hook.

Conclusion

While the initial study suggested that AI can slow down developers, the video argues that AI can significantly boost productivity when used correctly. The key is to use the right tools, follow disciplined workflows, provide ample context, and maintain human oversight. The video emphasizes the importance of upskilling with AI and using it to build new features rather than just modifying existing code. The use of custom hooks in Claude Code is presented as a practical way to enhance the AI's performance. The main takeaway is that AI is a powerful tool, but it requires skill and understanding to be used effectively.