Deepseek V3.2 (Speciale) & Mistral Large 3 (Fully Tested): The OGs of Open Models are BACK!

Key Concepts

• Deepseek V3.2: A new, non-experimental version of Deepseek's large language model, featuring a modified architecture.
• Mistral Large 3: A new model from Mistral, presented as state-of-the-art (SOTA) based on provided benchmarks.
• Deepseek Sparse Attention (DSA): A novel attention mechanism designed to reduce computational cost for large context lengths by focusing on the most relevant tokens.
• "Speciale" Model (Deepseek V3.2): A specialized variant of Deepseek V3.2 trained specifically for reasoning tasks.
• Mixture of Experts (MoE): An architectural approach where different "expert" sub-networks are activated for different parts of the input. Mistral Large 3 utilizes this, activating 41 billion parameters out of 645 billion.
• Context Length: The amount of information a model can process and retain at once.
• Computational Cost: The resources (processing power, memory) required to run a model.
• Benchmarks: Standardized tests used to evaluate and compare the performance of AI models.
• Tool Calling: The ability of a model to interact with external tools or APIs.
• Reasoning: The ability of a model to perform logical deduction and problem-solving.

Deepseek and Mistral New Model Releases

This video discusses two significant new open-source large language models: Deepseek V3.2 and Mistral Large 3. Both companies, Deepseek and Mistral, have a history of contributing to the open-source AI community, though Mistral has faced criticism for its licensing practices with larger models in the past.

Mistral Large 3 and Related Models

Mistral has launched new models, including Mistral Large 3, Mistral 314B, 8B, and 3B. These are presented as state-of-the-art (SOTA) based on benchmarks provided by Mistral.

• Architecture: Mistral Large 3 is described as a Mixture of Experts (MoE) model. It activates approximately 41 billion parameters out of a total of 645 billion.
• Compatibility: The architecture is based on Deepseek V3 and is compatible with most libraries that support it.
• Key Characteristic: A significant point highlighted is that Mistral Large 3 is a "non-reasoning model." It is characterized as a "raw and simple model" that does not perform any kind of reasoning.
• Performance Concerns: The presenter expresses skepticism about Mistral's confidence in their benchmarks, noting that the comparison models chosen suggest a lack of strong performance.

Deepseek V3.2 Release

Deepseek has released their "non-experimental" Deepseek V3.2, which introduces significant architectural changes.

• Deepseek Sparse Architecture (DSA): The core innovation is the adoption of DSA. This addresses the bottleneck of attention mechanisms in large language models, where computational cost explodes with increasing context length.
  • Mechanism: DSA uses a "lightning indexer" that acts like a spotlight, quickly scanning the context to identify the top k most important tokens relevant to a query, ignoring the rest. This contrasts with the standard transformer attention mechanism that attends to every token equally.
  • Benefit: This significantly reduces computational complexity while aiming to maintain the performance of dense models. It makes the model "incredibly cheap to run," even with long contexts up to 128,000 tokens.
• "Speciale" Variant: A key highlight is the "Speciale" model, a distinct checkpoint of Deepseek V3.2 specifically designed for reasoning.
  • Training Approach: The constraints on length penalties were relaxed during training, allowing the model to "think for as long as it needs to." This is a fundamental difference in training, not an inference-time parameter.
  • Availability: Both the general and "Speciale" variant weights are available on Hugging Face.

Performance Evaluation on Custom Benchmarks

The presenter shares their own benchmark results for both Mistral Large 3 and Deepseek V3.2 (non-reasoning variant).

Mistral Large 3 Benchmark Results

Mistral Large 3 performed poorly on several prompts:

• 3D Floor Plan: Failed to create a 3D floor plan for a 1,585 sq ft land.
• SVG Panda: Generated a panda, but the body was "finicky" and not a good generation.
• Pokeball in 3JS: Objects were misplaced, dimensions were incorrect, and it was not a good generation.
• Chessboard with Autoplay: Did not work at all.
• Kandinsky Style Minecraft Clone: Not usable and lacked quality.
• Majestic Butterfly: Described as "kind of fine," but not SOTA.
• Rust CLI Tool: Did not work.
• Blender Script for Pokeball: A failure.
• Math Questions: All failed.

The presenter concludes these results were "disappointing."

Deepseek V3.2 (Non-Reasoning Variant) Benchmark Results

The non-reasoning variant of Deepseek V3.2 also showed mixed results, with the presenter noting it performs worse on their benchmarks due to confusion, a recurring issue with previous Deepseek models.

• 3D Floor Plan: Not a 3D floor plan, but "gibberish and text."
• SVG Panda: Better than Mistral Large, but not SOTA.
• Pokeball in 3JS: "Quite good," with only the button missing.
• Chessboard: "Quite good," working seamlessly with sensible autoplay moves.
• Kandinsky Style Minecraft Clone: Did not work.
• Majestic Butterfly: "Not good," looking outdated.
• Rust CLI Tool: Did not work.
• Blender Script: Did not work.
• Math Questions: Not solved.
• Riddle: Solved "quite well."

Leaderboard Performance and Comparative Analysis

The video then discusses the performance of these models on leaderboards and offers a comparative perspective.

Leaderboard Scores

• Deepseek V3.2: Scores 11th position on the leaderboard, outperforming GPT 5.1 CEX and GLM. This is considered good, especially since it's based on an existing base model (Deepseek V3) with new experiments.
  • Reasoning Variant: The reasoning variant performed poorly, scoring much lower. It struggled to finish answers via API and produced buggy code, making it a "bad experience." The presenter states Deepseek's reasoning is "still very finicky and does more harm in coding than good."
• Mistral Large 3: Scores 27th position, which is deemed "kind of fine but not the best."
• Codestral Model: Stealth models on Kilo, Klein, and Rue (likely referring to Mistral's codestral model) showed some promise.

Presenter's Overall Opinion

The presenter's personal opinion is that GLM and Minimax are still superior, and Kimmy is also a great option. They suggest there isn't a strong need to switch to these new models for most users. However, they acknowledge that it's positive to see new, good, open models being released.

Conclusion

The release of Deepseek V3.2 and Mistral Large 3 brings new options to the open-source LLM landscape. Deepseek's V3.2, particularly with its DSA and "Speciale" reasoning variant, shows promise in addressing computational efficiency and reasoning capabilities, although the reasoning variant's practical application is still questioned. Mistral Large 3, while positioned as SOTA, demonstrated significant weaknesses in practical application and reasoning tasks during the presenter's evaluation. Despite the mixed results, the continued development of open models is seen as a positive trend.