The Future of Qwen: A Generalist Agent Model — Junyang Lin, Alibaba Qwen

Key Concepts

• Large Language Models (LLMs) & Large Multimodal Models: The core technology behind Quen, aiming for generalist AI.
• Instruction Tuning: Fine-tuning LLMs to follow instructions effectively.
• Reinforcement Learning (RL): Using rewards to train models to perform tasks better, especially reasoning.
• Hybrid Thinking Mode: Combining thinking (reasoning) and non-thinking (chatbot) behaviors in a single model.
• Dynamic Thinking Budget: Limiting the number of tokens a model can use for its "thinking" process.
• MOE (Mixture of Experts) Models: Models with many parameters but only activating a subset for each input, improving efficiency.
• Omni Model: A model that accepts and generates multiple modalities (text, vision, audio).
• WebDev: A feature that generates website code from simple prompts.
• Deep Research: A feature that conducts research and generates reports on specified topics.
• Inference Time Scaling: Improving model performance during inference by allowing it to interact with the environment and think iteratively.
• GUI Agent: An agent capable of interacting with graphical user interfaces.

Quen Overview and Resources

• Quen is a series of large language and multimodal models with the goal of building generalist AI.
• Quen Chat (chat.quen.ai): A chat interface to interact with the latest Quen models, including multimodal and omni models. Features include webdev and deep research.
• Quen Blog (quen.github.io): Technical details and updates on new releases.
• Open Source: Quen's code is available on GitHub, and checkpoints are on Hugging Face.

Quen 2.5 Max and Reinforcement Learning

• Quen 2.5 Max: Released before the Spring Festival, a strong instruction-tuned model.
  • Achieved competitive performance against models like Claude 3.5, GPT-4o, and DCB v3.3 on various benchmarks.
• Reinforcement Learning (RL) for Reasoning: RL significantly improves performance in reasoning tasks like math and coding.
  • Example: A 32 billion parameter model improved from 65% to 80% accuracy on a task after RL training.
  • Competitive performance in Triple Arena, ranking in the top 15 for a long time.

Quen 3: Latest LLM Release

• Quen 3 is the latest generation of LLMs, with multiple sizes of dense and MOE models.
• Flagship MOE Model: 235 billion parameters, activating 22 billion.
  • Competitive with top-tier models like Groq 3 mini, lagging slightly behind Gemini 1.5 Pro.
• Fast MOE Model: 30 billion parameters, activating 3 billion.
  • Outperforms Quen 32 billion in some tasks.
• Small Model: 4 billion parameters.
  • Distilled knowledge from larger models, showing competitiveness with Quen 2.5 72B.
  • Can be deployed on mobile devices.

Key Features of Quen 3

• Hybrid Thinking Mode: Combines thinking (reasoning) and non-thinking (chatbot) behaviors in a single model.
  • Thinking mode involves reflection and exploration before answering.
  • Non-thinking mode provides near-instant answers like a traditional chatbot.
  • Prompts and hyperparameters can control the model's behavior.
• Dynamic Thinking Budget: Limits the number of tokens used for the "thinking" process.
  • Performance increases with a larger thinking budget.
  • Example: Performance on a task increased from just over 40% to over 80% with a thinking budget increase from small to 32,000 tokens.
• Language Support: Supports over 119 languages and dialects, expanding from 29 in Quen 2.5.
• Agent and Coding Capabilities: Enhanced support for agents and coding, especially with the MCP framework.
  • Models can use tools during thinking, interact with the environment, and continue thinking based on feedback.
  • Example: A model can organize a desktop by thinking about which tools to use, using them, and then reporting the results.

Model Sizes and Open Weighting

• Multiple model sizes are available, including MOE and dense models.
• MOE Models: 30 billion (activating 3 billion) and 235 billion (activating 22 billion).
• Dense Models: Six different sizes.
• Smaller models (4 billion) can be deployed on mobile devices.
• 32 billion parameter models are strong and can be used for RL and local deployment.
• Open weighting of dense models, with plans to release more MOE models.

Multimodal Models

• Focused on vision language models.
• Quen 2.5 VL: Released in January, achieving competitive performance on vision language benchmarks like MMU, MathVista, and VQA.
• Quen VQA: Explores thinking capabilities for vision language models, showing inverse time scaling with larger maximum thinking length.
• Omni Model: Accepts text, vision (images and videos), and audio as inputs, and generates text and audio.
  • Based on a 7 billion LLM.
  • Achieves state-of-the-art performance in audio tasks for its size.
  • Surprisingly, performs better than Quen 2.5 VL 7 billion in vision language understanding tasks.
  • Needs improvement in language tasks, especially agent tasks.

Open Sourcing and Community

• Open sourcing is crucial for feedback and improvement.
• Various titled models in the open-source community, including LLMs and coders.
• Quen 2.5 Coders: Widely used for local development. Quen 3 Coders is in development.
• Quantized models are available in different formats (GUM, GBQ, AWQ, MOX).
• Apache 2.0 license for most models, allowing free use and modification.
• Supported by many third-party frameworks and API platforms.

Products and Features

• Building products for users to interact with Quen models.
• WebDev: Generates website code from simple prompts.
  • Examples: Creating a Twitter website or a sunscreen product introduction website.
  • Can also create cards from links and information.
• Deep Research: Writes reports on specified topics by making a plan, searching step-by-step, and writing parts iteratively.
  • Generates comprehensive reports that can be downloaded as PDFs.
  • Improving quality with reinforcement learning.

Future Directions

• Moving towards AGI and building good foundation models and agents.
• Pre-training: Still much room for improvement in training data, cleaning, multimodal data, synthetic data, and training methods (potentially using RL).
• Scaling Laws: Shifting focus to scaling compute in reinforcement learning and long-horizon reasoning with environment feedback.
• Context Scaling: Aiming for 1 million tokens context length this year, moving towards infinite context.
• Modality Scaling: Unifying understanding and generation across modalities (e.g., image understanding and generation).
• Training Agents: Moving from training models to training agents with RL and environment interaction.

Conclusion

Quen is actively developing and releasing advanced LLMs and multimodal models, with a strong emphasis on open sourcing, reinforcement learning, and building practical applications. The focus is shifting towards creating intelligent agents that can interact with the environment and perform complex tasks. The speaker emphasizes the importance of community feedback and collaboration in driving further progress.