The Llama 4 Herd - Open Source Won?

Llama 4: A Deep Dive

Key Concepts:

• Llama 4 (Scout, Maverick, Reasoning, Behemoth)
• Open-source AI
• Mixture of Experts (MoE)
• Context Length (10 million tokens, 1 million tokens)
• Multimodal (Image Understanding, Image Reasoning)
• Benchmarks (LM Arena ELO, Life Codebench, MMLU Pro, GPQA, Needle in a Haystack)
• Active Parameters vs. Total Parameters
• Licensing (700 million active users)

1. Llama 4 Overview and Model Variants

Meta AI is releasing Llama 4, aiming to establish open-source AI as the leading paradigm. The release includes multiple models:

• Llama 4 Scout: A fast, natively multimodal model with a nearly infinite 10 million token context length, designed to run on a single GPU. It has 17 billion active parameters (110 billion total) and uses 16 experts. It is positioned as the highest-performing small model in its class.
• Llama 4 Maverick: A workhorse model that outperforms GPT-4o and Gemini Flash 2 on benchmarks. It is smaller and more efficient than DeepSeek v3 while maintaining comparable text performance. It is natively multimodal, has 17 billion active parameters (400 billion total), uses 128 experts, and is designed for easy inference on a single host. It has a 1 million token context window.
• Llama 4 Reasoning: Details to be shared in the next month.
• Llama 4 Behemoth: A massive model with over 2 trillion total parameters and almost 300 billion active parameters, claimed to be the highest-performing base model, even before training completion.

2. Model Architecture: Mixture of Experts (MoE)

Llama 4 marks a shift from dense models to Mixture of Experts (MoE). This architecture is becoming prevalent in the industry, with Gemini and DeepSeek also adopting it. MoE models are more compute-efficient. Llama 4 Maverick demonstrates this, achieving the highest ELO score on the LM Arena leaderboard at a lower cost compared to other frontier models.

• Mixture of Experts (MoE): A neural network architecture where different parts of the network (experts) specialize in different types of data or tasks. A gating network determines which experts are activated for a given input.

3. Performance Benchmarks and Comparisons

Llama 4 Maverick is currently ranked second on the Chatbot Arena leaderboard, surpassing GPT-4o, Grok 3, and GPT-4.5 in user preference.

• Image Reasoning: Llama 4 Maverick achieves state-of-the-art performance in its class, comparable to Gemini 2.0 Flash and GPT-4o.
• Text-Based Benchmarks: While strong, Llama 4 Maverick is closely matched or slightly behind DeepSeek v3 on benchmarks like Life Codebench and MMLU Pro. It outperforms DeepSeek v3 on GPQA, but the differences are not significant.
• Coding: Only Life Codebench results are reported. Independent benchmarks like Sweetbench are needed for a comprehensive coding performance evaluation.
• Llama 4 Scout: Outperforms previous Llama versions, Gemma 7B, Mistral 7B, and Gemini 2.0 Flashlight on tested benchmarks.

4. Multimodal Capabilities and Long Context Handling

Llama 4 models possess image understanding and reasoning capabilities. They can answer questions based on input images and ground their answers within the image.

• Image Grounding: The ability to identify and reason about specific elements within an image.
• Long Context: Llama 4 Scout's 10 million token context window is particularly useful for retrieval systems. The "needle in a haystack" test demonstrates its ability to accurately retrieve information placed at various depths within the context window. Llama 4 Maverick also performs well within its 1 million token context window.

5. Practical Considerations and Limitations

• Hardware Requirements: Running Llama 4 Maverick requires an H100 GPU with at least 80GB of VRAM. Utilizing the full 10 million token context window of Llama 4 Scout demands significantly more GPU VRAM.
• Licensing: Companies with over 700 million monthly active users need a special license from Meta, which Meta can grant or deny. The "Built with Meta" attribution is required. This is the same license as Llama 2 and Llama 3.
• Open Weight vs. Open Source: Llama 4 is an open-weight model, meaning the model weights are available, but the training code and data are not.

6. Accessing and Testing Llama 4

• Online Platforms: Together AI and Groq offer access to Llama 4 Scout. Meta.ai allows interaction with the model after signing up.
• Self-Hosting: Model weights are available on Hugging Face. An H200 or B200 GPU can be used for faster performance.

7. Key Quotes

• "Our goal is to build the world's leading AI open-source it and make it universally accessible so that everyone in the world benefits." - Meta AI representative
• "I think that open- source AI is going to become the leading models and with Llama 4 this is starting to happen." - Meta AI representative

8. LM Arena ELO Score

The Llama model family has seen a significant jump in ELO score, from around 1270 to 1417, placing it just behind Gemini 1.5 Pro. This indicates a substantial improvement in user preference.

9. Conclusion

Llama 4 represents a significant advancement in open-weight models, particularly with its adoption of MoE architecture and long context capabilities. While hardware requirements and licensing terms pose some limitations, the release offers powerful tools for research and development. The coding abilities of the models, however, require further investigation through independent benchmarks. The trend towards larger models and longer context windows is likely to continue, shaping the future of AI development.