Gemma 3 architecture and design

Gemma 3 Model Architecture and Design: A Deep Dive

Key Concepts: Tokenizers, Model Sizes (1B, 4B, 12B, 27B), Quantization, Multimodal Capabilities, Multilingual Abilities, Long Context (128k tokens), Global Attention, Local Attention, Scaling, Finetuning.

1. Tokenizer Improvements

• Problem: Inadequate tokenizers can lead to splitting words into multiple tokens, especially for non-Latin languages, increasing sequence length and computational cost.
• Gemma's Approach (Gemma 2): Used a large tokenizer (256,000 tokens) for good multilingual ability and extended effective sequence length.
• Gemma 3 Update: Improved tokenizer coverage for 64 languages, including Hindi, Japanese, Korean, and Bengali. This enhances multilingual capabilities and allows for better finetuning on diverse languages.

2. Model Sizes and Scaling

• Model Sizes: Gemma 3 offers four sizes: 1B, 4B, 12B, and 27B. The 2B and 9B sizes from Gemma 2 were slightly increased to 4B and 12B to enhance multimodal and multilingual abilities. The 27B model size remains the same.
• Quantization-Aware Design: Model sizes were chosen considering their performance in quantized setups on popular hardware, optimizing for real-world deployment. Models are released in four quantization levels.
• 12B Sweet Spot: The 12B model is highlighted as a good balance between performance and cost.
• Predictable Scaling (4B-27B): All models from 4B to 27B share the same multimodal capabilities, extra-large context length, width-to-depth ratio, training regime, and data mixture (multilingual, code, math, STEM). This allows for seamless scaling up and down without unexpected behavior changes.
  • Example: Prototype with 4B, scale to 27B for product launch, and scale down to 12B for cost optimization.
• 1B Model (Blueberry): Specialized for tasks like text classification and rewriting short sentences. It lacks multimodal capabilities and is not focused on STEM or code.

3. Long Context Implementation

• Context Length Extension: Increased from 8,000 tokens in Gemma 2 to 128,000 tokens in Gemma 3.
• Practical Implications: Enables processing of 500 pages of text, 500 images, 8 minutes of video (1 frame per second), or an entire code base within a single prompt.
• Finetuning-Friendly Design: The long context recipe prioritizes simplicity for finetuning, avoiding complex approaches like YaRN or LongRoPE.
• Recipe:
  1. Tune the wavelength of global attention layers.
  2. For the last few percent of training, adjust the scale factor of global attention layers.
• Importance of Scale Factor: The scale factor adjustment is crucial for maintaining performance at longer sequence lengths (up to 128k tokens). Without it, performance degrades significantly beyond 32k tokens.
• Extended Context Beyond 128k: The 12B and 27B models can perform well up to 256,000 tokens, and the 27B model can even handle 500,000 tokens.

4. Inference Cost Optimization

• Attention Cost: Attention is a computationally expensive operation that scales quadratically with context length.
• Memory Savings: Gemma 3 12B requires approximately 26 GB of GPU memory at a context length of 128k, compared to 49 GB for a similar Llama-style model.
• Reduced Attention: Gemma 3 reduces attention cost by using fewer global attention layers.
• Global vs. Local Attention:
  • Global Attention: Attends over the entire sequence.
  • Local Attention: Attends over small chunks (e.g., a few sentences).
• Gemma 3 Architecture: Employs five local attention layers for each global attention layer. This significantly reduces inference cost without affecting model quality.
• Comparison to Llama: Llama-style models use only global attention layers. Gemma 2 had an interleaved one global and one local attention.

5. Key Arguments and Perspectives

• Simplicity in Long Context: The speaker emphasizes the importance of a simple and finetuning-friendly approach to long context, contrasting it with more complex methods.
• Cost-Effectiveness: The design choices in Gemma 3 prioritize cost-effectiveness, making it easier to deploy and run the models on available hardware.
• Scalability: The consistent architecture across different model sizes enables seamless scaling without surprises.

6. Conclusion

Gemma 3 introduces several architectural and design improvements focused on enhancing multilingual capabilities, extending context length, and optimizing inference cost. The key takeaways are the improved tokenizer, the predictable scaling across model sizes, the simple yet effective long context recipe, and the reduced attention mechanism that significantly lowers memory requirements. These advancements make Gemma 3 a more versatile, scalable, and cost-effective language model for a wide range of applications.