I tested DeepSeek vs. o3-mini for developers: Here’s what I found.

Key Concepts

• LLMs for Development: Using Large Language Models (LLMs) like DeepSeek R1, GPT-4o, and Claude 3.5 Sonnet to assist with software development tasks.
• Context Window: The amount of text (tokens) an LLM can process at once, impacting its ability to handle large projects.
• Reasoning Models: LLMs designed to think through problems step-by-step before generating code.
• Single-Shot Generation: The ability of an LLM to produce a complete solution (e.g., an entire application) in one attempt.
• Refactoring: Improving the structure and readability of existing code without changing its functionality.
• Test Generation: Automatically creating unit tests to verify the correctness of code.
• Token Cost: The price of using an LLM, measured in dollars per million tokens of input or output.
• CrewAI: A framework for automating tasks using AI agents.
• Streamlit: An open-source Python library for creating web applications for machine learning and data science.
• Ollama: A tool for running LLMs locally on your computer.

Model Performance Comparison: DeepSeek R1 vs. GPT-4o Mini

1. Core Stats: Performance, Price, and Context Window

• Performance (U-Bench Score):
  • DeepSeek R1, GPT-4o (reasoning models), and Claude 3.5 Sonnet have high scores (49+).
  • GPT-4o has a lower score (33), making it less suitable for coding tasks.
• Price (per million output tokens):
  • DeepSeek R1: $2
  • GPT-4o Mini: $4
  • Claude 3.5 Sonnet: $15 (7x more expensive than DeepSeek R1)
• Context Window:
  • GPT-4o (reasoning models) and Claude 3.5 Sonnet: 200,000 tokens (approximately 500 pages).
  • Output token limits:
    • GPT-4o (reasoning models): 100,000 tokens
    • DeepSeek and Claude: 8,000 tokens (more limited for large projects)
• Concrete Example:
  • Analyzing a 10,000-token file and generating a 2,000-token solution:
    • DeepSeek R1: $0.01
    • GPT-4o Mini: $0.02
    • Claude: $0.06

2. Test 1: Building a New Project from Scratch (Chat Interface with Ollama)

• Prompt: Create a ChatGPT-style website that works with Ollama to chat with local models.
• Methodology: Uniform prompt given to DeepSeek and GPT-4o Mini via web interface and Cursor IDE.
• GPT-4o Mini (Web):
  • Smaller input window due to web limitations.
  • Generates a single HTML file containing the entire application.
  • Functional website with model selection, but event listeners had errors.
• GPT-4o Mini (Cursor):
  • Utilizes full 100,000-token output window.
  • Generates separate HTML, CSS, and JavaScript files.
  • Fully functional chat interface with chat history.
• DeepSeek R1 (Web):
  • Limited by web interface, generates a single, shorter HTML file (291 lines of code).
  • Functional chat interface, but lacks features like chat history.
• DeepSeek R1 (Cursor):
  • Failed to build the application, only produced a single JavaScript file.
  • Struggled with large context windows and multi-step tasks.
• Claude:
  • Failed to build the application, produced disconnected code snippets.
• Conclusion: GPT-4o Mini in Cursor performed best for building a new project from scratch.

3. Test 2: Building a New Feature in an Existing Application (CrewAI Chat UI)

• Feature: Create a Streamlit-based web UI for the CrewAI chat feature.
• Methodology: Pass a prompt referencing multiple code files to DeepSeek and GPT-4o Mini in Cursor.
• GPT-4o Mini (Cursor):
  • Generated new UI files and updated existing command-line files.
  • Struggled with understanding how CrewAI projects are structured for end-users.
  • Had issues with Streamlit state management and proper crew execution.
  • Required 20+ iterations of feedback and corrections.
  • Ultimately successful, but with high friction.
• DeepSeek R1 (Cursor):
  • Generated initial code changes but didn't specify where to insert them.
  • Needed reminders to create the UI application.
  • After initial guidance, performed well and generated high-quality code.
  • Required 8-10 iterations, fewer than GPT-4o Mini.
  • Created a more polished Streamlit UI without experimental feature issues.
• Conclusion: DeepSeek R1 was the winner for building new features, requiring more developer involvement but achieving results faster.

4. Test 3: Refactoring Existing Code and Generating Tests

• Task: Refactor a CrewAI chat file and generate unit tests.
• Methodology: Pass a prompt and code files to DeepSeek and GPT-4o Mini.
• GPT-4o Mini:
  • Refactored the code, added comments, and moved translations to an internationalization file.
  • Messed up a type in the generated tests, requiring further refinement.
  • Overall, a successful refactoring and test generation.
• DeepSeek R1:
  • Failed the refactoring task by removing important code and breaking the application.
  • Added exception handling but only partially implemented it.
  • Successfully added translations to the internationalization file and created tests.
  • The broken code outweighed the successful aspects.
• Conclusion: GPT-4o Mini was the winner for refactoring and test generation due to DeepSeek R1's critical failure.

5. Final Recommendations and Workflow

• Overall Winner: GPT-4o Mini (with caveats).
  • Faster and more consistent results.
  • Larger context window for bigger changes.
• Biggest Losers: Claude 3.5 and GPT-4o (original).
  • Claude 3.5 is too expensive.
• Recommended Workflow: "Go Wide, Then Deep"
  1. GPT-4o Mini: Use to brainstorm, explain the task, and generate a V1 of the code.
  2. DeepSeek R1: Use to refine individual functions and perform targeted code edits.
• Community Engagement: Encourage users to share their experiences and alternative approaches.

6. Notable Quotes

• "Deep seek is phenomenal... but it needed like additional reminding, whereas 03 mini was just more consistent in the long run."
• "Deep seek required more brain power but got there way faster."
• "If you're trying to build an entire project from scratch, stay away from Claude."

7. Technical Terms

• U-Bench Score: A benchmark score for evaluating the coding performance of LLMs based on solving issues in a GitHub repository.
• Tokens: Units of text used by LLMs for processing.
• Context Window: The maximum number of tokens an LLM can process at once.
• Streamlit: A Python library for creating web applications for machine learning and data science.
• CrewAI: A framework for automating tasks using AI agents.
• Ollama: A tool for running LLMs locally.

8. Synthesis/Conclusion

GPT-4o Mini and DeepSeek R1 are powerful tools for developers, each with strengths and weaknesses. GPT-4o Mini excels at generating initial code and handling larger projects, while DeepSeek R1 is better for targeted code edits and optimization. The "go wide, then deep" approach leverages both models for maximum efficiency. Claude 3.5 is not recommended due to its high cost and limited output. Developers should carefully consider the specific task and model characteristics to choose the best tool for the job.