Deepdive into Gemma 3

Key Concepts:

• Gemma 3: A new family of open-source language models.
• Model Sizes: 1B, 4B, 12B, and 27B, each with varying capabilities.
• Quantization: Techniques (bfloat16, float8, 4, q4_0) to reduce model size and memory footprint.
• Context Length: The amount of text a model can consider at once (increased to 128k tokens for larger models).
• Multilinguality: Support for over 140 languages.
• Multimodality: Ability to process interleaved image and text inputs.
• Function Calling: Generating programming calls as part of an AI agent workflow.
• UniMax Sampling: A distribution used for selecting languages during training.
• Vision Encoder: A SigLIP variant used to process images.
• Interleaved Attention: Alternating between local and global attention layers.
• RoPE (Rotary Positional Embeddings): Positional embeddings used in the model.
• Instruction Tuning: Shaping the model's behavior for specific tasks.
• Zero-Shot Performance: Ability to perform tasks without fine-tuning.
• Pan and Scan: A technique to process images with skewed aspect ratios.

1. Introduction of Gemma 3

• Cassidy Hardin introduces Gemma 3, the latest family of open-source models, emphasizing their capability, portability, and responsibility.
• Gemma 3-27B-IT ranked in the top 10 models on LMSYS Chatbot Arena, outperforming models 15 times its size.

2. Gemma 3 Model Sizes and Capabilities

• The Gemma 3 family includes four sizes:
  • 1B: Lightweight text model for small applications (less than 700M memory footprint when quantized for q4_0). Specializes in English and multilingual performance with impressive STEM performance. 32k context length.
  • 4B: Balances flexibility and performance with multimodal support.
  • 12B: Strong language capabilities for complex tasks.
  • 27B: Most sophisticated model for intricate tasks and demanding applications.
• Pre- and post-train checkpoints are released for each model size, with versions quantized for bfloat16, float8 and 4, and q4_0.

3. Technical Improvements in Gemma 3

• Longer Context Length: 4B, 12B, and 27B models support 128k tokens (16 times longer than Gemma 2).
• Expanded Multilinguality: Supports over 140 languages.
• Multimodality: Supports interleaved image and text on input with text-only output.
• Function Calling: Allows creating natural language interfaces for programming APIs.

4. Multilinguality Advancements

• Doubled the amount of multilingual data for pre- and post-training.
• Utilized a UniMax sampling distribution for language selection.
• Robert Dadashi mentions Gemma 3-27B model is now leading all open-weights model in French and in Spanish on the LMSYS Chatbot Arena.
• Robert Dadashi provides an example in French to demonstrate the model's understanding of language, cultural context, and formatting requirements.

5. Multimodal Capabilities

• Gemma 3 supports interleaving text and images on input, outputting text.
• Uses a SigLIP variant vision encoder tailored to specific needs.
• Aishwarya Kamath highlights that Gemma 3 is one model that supports any resolution, unlike PaliGemma which provides a separate model for three input resolutions.
• Aishwarya Kamath mentions Gemma 3 is great at zero shot performance and can be used out of the box on a wider variety of tasks without fine-tuning on the specific examples, unlike PaliGemma models which are geared towards task transfer through fine-tuning on specific examples.
• Aishwarya Kamath mentions Gemma 3 models are instruction tuned, unlike PaliGemma.
• Aishwarya Kamath mentions Gemma 3 supports fully interleaved image and text, unlike PaliGemma.
• Aishwarya Kamath mentions the 4B model is 10 times cheaper to transfer while outperforming the larger-size PaliGemma models on high-resolution tasks.
• Aishwarya Kamath explains how multimodal inputs are processed, including image resizing, vision encoder processing, and token generation.
• Aishwarya Kamath illustrates the attention pattern used in Gemma 3, which is fully causal on text and bidirectional on images.
• Aishwarya Kamath mentions Gemma 3 achieves great performance on reasoning, math, document understanding, and video understanding.
• Aishwarya Kamath explains the pan and scan approach to process images with skewed aspect ratios.

6. Architecture Improvements

• Shreya Pathak mentions the pre-training recipe remains similar to Gemma 2, with some updates to aid the new capabilities that are being added.
• Shreya Pathak mentions the models are trained with a frozen visual encoder.
• Shreya Pathak mentions Gemma 3 is trained at 32k from the start, and in the final stages of training, the context length is further extended, allowing the model to support up to 128k tokens.
• Attention pattern updated to five local layers for every global layer.
• Sliding window size updated to 1024 (512 for the 1B model).
• RoPE config updated with increased frequency for global layers (1 million) and rescaling in the final training phase.

7. Post-Training and Instruction Tuning

• Léonard Hussenot explains post-training shapes the model's behavior for various capabilities.
• Gemma 3 outperforms Gemma 2 in code, chat, and math tasks.
• Gemma 3-4B is almost on par with Gemma 2-27B.
• Instruction-tuned models require specific formatting sequences (beginning-of-sequence token, turn markers).
• Specific prompting improves performance (e.g., "only output Python code").
• System instructions should be placed in the first user prompt.

8. Multimodal Use Cases and Examples

• Geoffrey Cideron presents examples of multimodal conversations with Gemma 3:
  • Reasoning: Computing the diagonal of a rectangle drawn on a whiteboard.
  • LaTeX Formula Generation: Transcribing a math equation from an image.
  • Multilingual Multiturn: Understanding a bus ticket from Prague and answering questions in multiple turns.
  • Plot Analysis: Analyzing a plot with two functions and answering questions about their properties.

9. Conclusion

• Gemma 3 represents a significant advancement in open-source language models, offering improved performance, multilinguality, multimodality, and longer context lengths.
• The models are designed to be accessible and useful for a wide range of applications.
• Google is committed to further improving Gemma's multilingual abilities.