DeepSeek V3.1: Bigger Than You Think!

DeepSeek 3.1 Model Analysis: A Minor Upgrade with Major Implications

Key Concepts:

• DeepSeek 3.1: A new open-weight language model from DeepSeek AI.
• Hybrid Inference Model: A model that combines reasoning (thinking) and non-reasoning capabilities into a single architecture.
• Token Efficiency: The ability of a model to achieve a certain level of performance while using fewer tokens.
• Agentic Tasks: Tasks that require the model to use tools and perform multi-step actions, often involving coding.
• Post-Training: Additional training applied to a pre-trained model to improve its performance on specific tasks.
• Open-Weight Model: A model whose weights are publicly available, allowing for greater accessibility and customization.
• SWE-bench Verified: A coding benchmark used to evaluate the performance of language models on software engineering tasks.
• API Precision: The numerical precision (e.g., 8-bit floating point, 4-bit) used when serving a model through an API, which can affect performance.

1. Introduction and Overview of DeepSeek 3.1

The video discusses the new DeepSeek 3.1 model, characterizing it as more than just a minor upgrade to V3. It is considered potentially the best open-weight model currently available. The model weights were released without initial documentation, but more information has since emerged.

2. Architecture and Training

• Hybrid Inference: DeepSeek 3.1 is a hybrid inference model, merging thinking and non-thinking capabilities into one. This eliminates the need for separate V3 and R1 models and likely cancels the development of an R2 model. The next iteration is expected to be 3.2, 3.5, or V4.
• Training Data: The model is built upon the V3 base model used for the R1 reasoning model, with an additional 800 billion tokens of pre-training.
• Focus on Tool Usage: The model is specifically post-trained for strong tool usage and function calling, making it suitable for multi-step agent tasks, agentic coding, and coding IDEs.
• Entropic API Support: The model supports the Entropic API, making it a potentially cost-effective option for use within cloud code.

3. Performance Benchmarks and Comparisons

• Significant Upgrade: DeepSeek 3.1 represents a substantial upgrade compared to previous DeepSeek versions (R1 and V3).
• SWE-bench Verified: Shows almost a 50% performance increase on SWE-bench verified compared to previous versions.
• SUB-bench Multilingual: Demonstrates even more significant performance improvements on SUB-bench multilingual.
• Comparison to Proprietary Models: While it may not surpass state-of-the-art proprietary models like GPT-4, it is considered state-of-the-art for open-weight models.
• Sonnet 4 Comparison: The non-reasoning version lags behind Sonnet 4 on some benchmarks, but the thinking version surpasses it on others. The SWE-bench verified results for the thinking version are not yet available.

4. Token Efficiency and Cost Implications

• Token Efficiency: DeepSeek 3.1 is more token-efficient than previous versions, meaning it uses fewer tokens to achieve the same level of performance.
• Cost Savings: Token efficiency translates to lower costs, as users pay per token.
• Comparison to Other Models: DeepSeek 3.1's token efficiency is comparable to OpenAI models, while Enthropic models tend to be less token-efficient.
• Gemini 2.5 Pro vs. 03 High: Gemini 2.5 Pro requires almost three times more tokens than 03 High to achieve similar performance on a specific benchmark, despite similar pricing.
• Tokens are Getting More Expensive: Despite the theoretical decrease in cost per token, reasoning models are generating more tokens, leading to higher overall costs. This is impacting companies like Cursor and Cloud Code.

5. Pricing and API Considerations

• Competitive Pricing: DeepSeek 3.1 offers competitive pricing, almost half a cent per million tokens without caching and $1.70 per million tokens with caching.
• API Stability: Concerns exist regarding DeepSeek's ability to maintain API stability, based on past experiences with R1.
• Open-Weight Advantage: As an open-weight model, it will be hosted by multiple providers, offering users more options.
• API Precision Impact: The model is trained in 8-bit floating-point precision, and serving it in lower precision (e.g., 4-bit) can degrade performance. Users should compare performance across different API providers to ensure they are getting the full potential of the model.

6. Artificial Analysis Intelligence Index

• Incremental Improvement: DeepSeek's own analysis shows an incremental improvement in the artificial intelligence index, with a score of 60 in Disney mode, up from R1's score of 59.
• Benchmark Limitations: Benchmarks are often indirectly included in training data, which can skew results.
• Hybrid Reasoning: The move to a unified hybrid reasoning model mirrors the approach taken by OpenAI, Anthropic, and Google.
• Tool Calling Strategy: Suggests using the reasoning mode for planning and the non-reasoning mode for executing tool calls, creating a hybrid approach for agentic systems.

7. Quick Test and Demonstration

• Bouncing Wall Problem: A modified version of the bouncing wall within a hexagon problem was used to test the model.
• Initial Solution: The model successfully generated a working solution on the first try after thinking for over 3 minutes.
• Subsequent Request: A subsequent request to add user interaction (changing rotation direction and explosion on click) resulted in a partial implementation. The rotation change worked, but the explosion behavior was not as expected.

8. Conclusion

Despite being labeled a minor upgrade, DeepSeek 3.1 is a significant release, potentially the best open-weight model available. Its hybrid architecture, improved token efficiency, and competitive pricing make it a compelling option for various applications, particularly those involving agentic tasks and coding. However, users should be mindful of API stability and potential performance variations due to different API providers using different precision levels. The model's performance on real-world tasks, especially in agentic coding IDEs, remains to be fully explored.