Qwen3 with INSANE Performance. It BEATS DeepSeek R1!

Quen 3 Open-Weight Model: Summary

Key Concepts:

• Quen 3: Open-weight language model family.
• Mixture of Experts (MoE): Model architecture with multiple sub-networks.
• Thinking Mode: Step-by-step reasoning process.
• Non-Thinking Mode: Quick response for simpler questions.
• Knowledge Distillation: Training smaller models from a larger model.
• Tool Calling (Agentic Behavior): Ability to use external tools.
• MCP: (Not explicitly defined in the transcript, but mentioned as a key feature).
• Chain of Thought (CoT) Reasoning: Training technique to improve reasoning.
• Reinforcement Learning (RL): Training technique to optimize model behavior.

1. Model Overview and Performance:

• Quen 3 is an open-weight model family that includes both dense and Mixture of Experts (MoE) models.
• The models are released under the Apache 2.0 license, allowing for commercial use.
• Quen 3 MoE models outperform OpenAI 01, OpenAI 03 Mini, and Deepseek R1.
• Model sizes range from 6 billion to 32 billion parameters for dense models.
• MoE models include a 30 billion parameter model with 3 billion active parameters and a 235 billion parameter model with 22 billion active parameters.

2. Key Features:

• Hybrid Thinking:
  • Thinking Mode: Enables step-by-step reasoning for complex problems.
  • Non-Thinking Mode: Provides quick responses for simpler questions.
  • Flexible control based on task requirements.
  • Stable and efficient thinking budget control.
• Multilingual Capability: Supports 119 languages with and without thinking mode. Thinking mode improves accuracy.
• Tool Calling (Agentic Behavior): Supports tool calling, enabling agentic behavior.
• MCP Support: Includes support for MCP (unclear what MCP refers to, but it's highlighted as a key feature).

3. Training Process:

• Trained on two times more data than Quen 2.5, including web data, PDF-like documents, and synthetic data generated using Quen 2.5.
• Three-Stage Training:
  1. Basic Language Skill: Initial training for fundamental language abilities.
  2. Coding Reasoning: Training focused on coding and reasoning skills.
  3. High-Quality Long Context Data: Training with long-context data to improve understanding and generation.
• Post-Training Process:
  1. Taught the base model with Chain of Thought (CoT) reasoning.
  2. Reasoning reinforcement learning.
  3. Thinking mode fusion.
  4. General reinforcement learning.
• Smaller models, including the MoE versions, were created from the 32 billion parameter dense model using knowledge distillation.

4. Usage and Implementation:

• Quen 3 can be used via Hugging Face Transformers, SGLang, or VLLM.
• Thinking mode is enabled by default.
• To explicitly enable thinking mode, add "/think" to the end of the prompt.
• To disable thinking mode, add "/no think" to the end of the prompt.

5. Examples and Demonstrations:

• Quen Chat Demo:
  • Available in Quen Chat with both MoE and smaller versions.
  • Example 1: Counting the number of "O"s in "strawberry" with extra "R"s. The model correctly identified five "O"s.
  • Example 2: Generating 10 sentences ending with "apple." The model failed as the sentences did not end with "apple."
  • Example 3: Asking the model to count the number of words in its next response. The model provided an incorrect count.
  • Example 4: Solving a Josephus permutation problem in Python. The model successfully generated correct code.

6. Notable Quotes/Statements:

• (Implied) "More thinking, more accurate" - The transcript suggests that using thinking mode leads to more accurate responses.

7. Technical Terms and Concepts:

• Open-Weight Model: A language model whose weights are publicly available.
• Mixture of Experts (MoE): A model architecture that combines multiple sub-networks (experts) to handle different types of inputs or tasks.
• Chain of Thought (CoT) Reasoning: A technique where the model is trained to generate a sequence of intermediate reasoning steps before providing the final answer.
• Knowledge Distillation: A technique for training a smaller model (student) to mimic the behavior of a larger, more complex model (teacher).

8. Logical Connections:

• The training process is designed to build upon basic language skills, then add coding and reasoning abilities, and finally refine the model with long-context data.
• Knowledge distillation is used to create smaller, more efficient models from the larger, more capable 32 billion parameter model.
• The thinking mode feature is directly related to the chain of thought reasoning training, allowing the model to explicitly show its reasoning process.

9. Synthesis/Conclusion:

Quen 3 is a promising open-weight language model family that offers a range of models with varying sizes and capabilities. Its Mixture of Experts architecture and hybrid thinking mode, combined with a comprehensive training process, allow it to achieve strong performance on a variety of tasks. While some examples in the demo showed failures, the model demonstrated impressive reasoning and coding abilities in other cases. The availability of the models on Hugging Face and the code on GitHub makes Quen 3 accessible for research and commercial applications.