Welcome to the Gemmaverse

Key Concepts:

• Gemma: A family of open-source models developed by DeepMind.
• Gemmaverse: The ecosystem surrounding the Gemma models, including developers, researchers, and community members.
• LLM: Large Language Model.
• Fine-tuning: Adapting a pre-trained model to a specific task or dataset.
• Quantization: Reducing the precision of a model's weights to decrease its size and improve performance.
• Context Length: The amount of text a model can process at once.
• System Prompts: Instructions given to a model to guide its behavior.
• LMCC: Language Model Capabilities Chart.
• LM Arena: A platform for evaluating language models.
• Ollama: A tool for running language models locally.
• Hugging Face: A platform for sharing and using machine learning models.
• vLLM: A high-throughput and memory-efficient inference engine for LLMs.
• Unsloth: A library for fast and memory-efficient fine-tuning of LLMs.
• LlamaIndex and LangChain: Frameworks for building applications with LLMs.
• DRL: Deep Reinforcement Learning.
• SEA-LION: A fine-tuned Gemma model for Southeast Asian languages.

Gemma's First Anniversary and Gemma 3 Release

• Omar Sanseviero, DevRel lead for Gemma and open models at DeepMind, celebrates the first anniversary of Gemma and the release of Gemma 3.
• He acknowledges the diverse open-source ecosystem present, including representatives from Cohere, Hugging Face, Ollama, NVIDIA, EleutherAI, AMD, and others.
• The Gemmaverse has grown significantly over the past year, with multiple model releases aimed at enabling developers and researchers to push the boundaries of open science and machine learning.

Gemma Ecosystem Growth and Community Contributions

• Since the initial Gemma release, DeepMind has released ShieldGemma, DataGemma, Gemma Scope, PaliGemma, and other models.
• Examples of community contributions include:
  • SEA-LION: AI Singapore's fine-tuned Gemma model for 11 Southeast Asian languages, integrated into AI voice assistants used by 38 million monthly active users in Indonesia.
  • Institute for Computer Science, AI, and Technology (Bulgaria): Developed a GPT-based LLM using Gemma 2 for Bulgarian and English.
• Over 60,000 Gemma-based models have been created on Hugging Face, showcasing diverse use cases, fine-tuning techniques, and applications in areas like embeddings, interpretability tools, and quantization.
• Applications range from Korean conversational models to foundations for image generation, audio understanding, and retrieval augmented generation for languages like Arabic.
• Hugging Face has seen consistent growth in Gemma downloads and usage over the past year.

Accessibility and Integration with Open Source Tools

• DeepMind has collaborated with partners like Ollama, vLLM, Hugging Face, and Unsloth to make Gemma accessible and easy to use within the open-source ecosystem.
• Ollama allows users to download and run Gemma 3 with a single command: Ollama run gemma3.
• Integrations with tools like LlamaIndex and LangChain enable developers to leverage Gemma in various applications.
• Hugging Face transformers and DRL libraries allow for quick model loading and fine-tuning in a few hours with limited data.

Community Feedback and Gemma 3 Improvements

• DeepMind actively sought community feedback on Gemma 2 through platforms like Reddit (LocalLLaMA) and developer events.
• Key improvements in Gemma 3 based on feedback:
  • Increased context length from 8,000 to 128,000 tokens.
  • Addition of a smaller 1B model.
  • Quantized-aware training checkpoints for preserving quality during quantization.
  • Support for image and video input.
  • Support for over 140 languages.
  • Improved instruct capabilities and support for system prompts.

Gemma 3 Performance and Benchmarks

• Gemma 2 was well-received by the community, demonstrating strong adoption and excitement.
• Gemma 3 achieves competitive performance, as indicated by LMCC scores (which increased by one point) and LM Arena scores.
• Gemma 3 performs strongly across different languages, ranking as a top model for French and Spanish.

Conclusion

• Gemma 3 is welcomed into the Gemmaverse, offering new capabilities and improvements based on community feedback.
• DeepMind is excited to see the applications, models, and contributions that the community will create using Gemma 3.
• The speaker expresses enthusiasm for continued collaboration and future developments in the Gemma ecosystem.