DeepSeek‑r1, o3‑mini & Gemini Flash 2.0: Which model should you use in your AI Agents?

Key Concepts

• AI Agents
• Large Language Models (LLMs)
• Context Window
• Instruction Following
• Tool Calling
• Retrieval-Augmented Generation (RAG)
• Token Usage
• Cost Efficiency
• Model Performance Benchmarking

Model Breakdown and Core Stats

The video begins by outlining the current landscape of AI models and the need to identify the best models for AI agents. It focuses on five key models:

1. Gemini 2.0 Flash: Boasts a 1 million token context window, making it suitable for handling large amounts of information, potentially eliminating the need for RAG in some cases. It's also noted for being cheap, but has limited output tokens.
2. GPT-3.5 Turbo (03 Mini): Described as a balanced model with a 200,000 input token and a 100,000 output token limit. It's considered a good developer model due to its intelligence and efficiency.
3. DeepSeek R1: An open-source model known for being the cheapest and smartest.
4. Claude 3.5 Sonnet: A widely used model by AI developers, but criticized for being expensive compared to its performance. It's mentioned as being seven times more expensive than DeepSeek.
5. GPT-4.0: Previously the standard for AI agents, but now facing competition from newer models.

The initial assessment suggests that GPT-3.5 Turbo (03 Mini) might be the best model on paper.

Test 1: Instruction Overload

This test evaluates the models' ability to follow complex instructions.

• Methodology: The models are tasked with writing a newsletter with specific formatting rules. A "crew" of three agents is used: a synthesizer, a newsletter writer, and a newsletter editor.
• Process:
  1. Synthesizer Agent: Processes a "brain dump" to create an outline and subject line, following detailed instructions and examples.
  2. Newsletter Writer Agent: Transforms the outline into a newsletter, adhering to a specific template and instructions.
  3. Newsletter Editor Agent: Reviews the generated newsletter to ensure it follows all rules.
• Evaluation Metrics: Time to complete, token usage, price, and whether the model passed or failed. A side-by-side comparison of the generated newsletters is also conducted.
• Results:
  • All models successfully generated a newsletter.
  • DeepSeek models (V3 and R1) took a long time (around 10 minutes). DeepSeek V3 struggled significantly.
  • Claude 3.5 Sonnet was expensive and inefficient, with excessive token usage due to repeated recalls. It failed due to struggling with word count and newsletter length.
  • GPT-4.0 performed well, completing the task quickly (under 2 minutes) and generating a good result, but was relatively pricey.
  • GPT-3.5 Turbo (03 Mini) excelled, being fast and efficient with minimal token usage.
  • Gemini 2.0 Flash was fast and cheap but exhibited some issues with repeating cycles.
• Newsletter Quality:
  • GPT-3.5 Turbo (03 Mini) produced the best newsletter, closely following instructions and requiring minimal changes.
  • Gemini 2.0 Flash performed well but didn't number everything and added extra content.
  • DeepSeek R1 had good formatting but didn't adhere to the core topic.
  • GPT-4.0 failed to add a subject line and didn't number the types of luck.
  • Claude 3.5 Sonnet was expensive and didn't follow instructions well, adding irrelevant information.

Test 2: Tool Hell

This test assesses the models' ability to use multiple tools in the correct order with the right parameters.

• Methodology: The models are given access to five tools: getting the current date, weather, Nvidia stock price, latest news on Elon Musk, and a square root calculator. They are instructed to use these tools in a specific order and generate a song in the style of "Twinkle Twinkle Little Star" using the information gathered.
• Process: The agent must call the tools in the specified order, passing the output of one tool as input to the next, and then use all the results to generate a poem.
• Evaluation Metrics: Time to complete, token usage, price, and pass/fail status.
• Results:
  • DeepSeek V3 failed due to poor tool calling abilities.
  • DeepSeek R1 was inconsistent, with tool calling working only 20% of the time.
  • Gemini 2.0 Flash called all tools correctly but failed to properly use the results in the final output.
  • OpenAI models (GPT-4.0 and GPT-3.5 Turbo) and Claude performed well with no issues.
  • GPT-4.0 was surprisingly fast.
• Final Output Quality:
  • Gemini 2.0 Flash called the tools but didn't use the results effectively in the final output.
  • GPT-4.0 and GPT-3.5 Turbo (03 Mini) produced excellent results, incorporating all the information correctly.
  • DeepSeek models failed.
  • Claude model worked but was expensive.

Test 3: Needle in a Haystack (RAG)

This test evaluates the models' ability to identify specific information within a large amount of data.

• Methodology: The models are given a large amount of random data with a single key piece of information hidden within it. They are then asked a question that requires them to find that specific piece of information.
• Process: The agent must sift through the large dataset to find the answer to the question: "What's the favorite thing Brandon Hancock likes to do?" The key information is: "Brandon Hancock makes YouTube videos for about AI for developers and entrepreneurs his favorite thing is when viewers just like you like And subscribe to the channel."
• Evaluation Metrics: Pass/fail status.
• Results:
  • DeepSeek V3 and Claude 3.5 Sonnet failed, unable to utilize their full context window effectively. They struggled with single-shot responses and inefficient token usage due to repeated recalls.
  • GPT-4.0 passed, processing 125,000 tokens and providing the correct answer. It was also the fastest.
  • GPT-3.5 Turbo (03 Mini) passed, processing 190,000 tokens with only slightly longer execution time.
  • Gemini 2.0 Flash excelled, processing over 900,000 tokens at a very low cost (8 cents) and providing a single-shot response.
• Key Observation: Gemini 2.0 Flash is highly effective and cost-efficient for data-intensive RAG applications.

Conclusion

GPT-3.5 Turbo (03 Mini) emerges as the best overall model for AI agents due to its well-rounded performance across all tests. However, the video highlights specific use cases where other models excel:

• Chatbots and High-Query Projects: Gemini 2.0 Flash is recommended due to its low cost and large context window.
• Process-Based Agents (No Tools): DeepSeek R1 is a viable option if cost is a primary concern and time is not critical.

The video concludes with reminders about the availability of free source code, access to a community of AI developers, and other AI-related content on the channel.