What's new in the Gemmaverse

Key Concepts

• Gemma: Google's open family of open models based on Gemini technology.
• Parameters: Variables in a model's mathematical equation (e.g., 'a' and 'b' in 'a*x + b*x'). More parameters generally mean a larger, more complex model.
• Multimodal: Models that can process multiple types of data, such as text and images.
• Context Window: The amount of text or data a model can process in a single prompt or conversation. A larger context window allows for longer conversations and more complex tasks.
• Quantization: A technique to reduce the precision of model parameters (e.g., from 16 bits to 8 bits or 4 bits), making the model smaller and faster but potentially sacrificing some accuracy.
• Pre-training: Training a model from scratch on a large dataset to predict the next token.
• Post-training: Fine-tuning a pre-trained model for specific tasks, such as improving conversational abilities or adapting to a particular domain.
• Safety Classifier: A model that classifies the safety of content, such as images, based on a given prompt or policy.

Gemma Overview and Philosophy

• Google DeepMind aims to solve intelligence and believes collaboration is key.
• They publish research and open models to benefit everyone.
• Gemma is designed to be accessible and responsible, prioritizing both benefiting and protecting people.
• Gemma models are available in various sizes and flavors.
• Launched last year, with over 150 million downloads to date.

Gemma Model Variants and Tools

• PolyGemma: First Gemma model with vision capabilities, allowing image-based questions.
• Shield Gemma: A safety classifier to enhance the responsibility of AI systems.
• Gemma Scope: Tooling to analyze model layers and understand reasoning behind results.
• Gemma.cpp: C++ implementation for running Gemma models on CPU only, enabling use on devices without strong GPUs.
• Gemma 3: Most capable model at the time of the presentation, available in 1B, 4B, 12B, and 27B sizes.
• Gemma 3N: Newest model optimized for on-device performance, requiring as little as 2GB of RAM. It is the base model for Gemini Nano.

Gemma 3 Features and Improvements

• Multimodal: Supports image input and multi-turn conversations with images.
• Multilingual: Supports over 140 languages.
• Larger Context Window: Increased from 8K to 32K (1B model) and 128K (4B, 12B, 27B models).
• The 27B model can still produce good results even with a 512K context window, although it is benchmarked up to 128K.

Examples of Gemma 3 in Action

• Analyzing an image of a ticket to extract the price and validity period.
• Translating a multilingual sign and providing context about its location.

Community Feedback and Model Personality

• Community feedback is crucial for improving Gemma.
• Improvements based on feedback include larger context windows, smaller models (1B), quantization, multimodal capabilities, and multilingual support.
• Gemma models have a warm, approachable personality, reflected in high LM Marina scores.
• The 27B model is a top-performing open model, requiring only one GPU to run.
• Gemma performs well in multiple languages, ranking high in French, Spanish, Japanese, and German.

Gemma 3N Details

• Optimized for on-device performance.
• Expands capabilities with audio understanding, allowing direct voice interaction.
• Available in 4B and 2B sizes.

Gemma 3N Demo

• A video showcases Gemma 3N running locally on a device, performing tasks such as:
  • Describing a replica of a Virgin Galactic spaceship.
  • Saving notes to Google Keep.
  • Suggesting caching layers to improve speed.
  • Writing a short poem.
  • Calculating the square footage of a room.
  • Identifying a cartoon drawing of Einstein.

Building with Gemma: Tools and Ecosystem

• AI Studio: Easiest way to get started, allowing users to test prompts and deploy models in Google Cloud Run.
• Google Cloud Run: Allows for automatic scaling of Gemma deployments.
• Kaggle: Platform for AI builders to discover models, share knowledge, and participate in challenges.
• Ollama: Tool for running Gemma models locally with a simple command.
• Keras: Provides high-level APIs for prototyping and building with Gemma models.
• Jax: Gemma Python library with high-level abstractions for running Gemma models.
• Hugging Face Transformers: Library with a pipeline class for text generation using Gemma models.

Quantization Details

• Each parameter is stored with a specific precision (e.g., Bfloat16 = 16 bits per parameter).
• Rough estimate: 2GB of VRAM required per billion parameters.
• Quantization reduces precision to decrease model size, but can degrade quality.
• Quantization Aware Training (QAT) makes models more robust to quantization, preserving quality even at lower precisions (e.g., INT4).

Gemmaverse: Pre-training and Post-training

• LLM training involves pre-training (training from scratch) and post-training (fine-tuning).
• Google releases both pre-trained and post-trained Gemma checkpoints.
• Post-training techniques include RLHF, Qura, PEF, and fine-tuning.
• Fine-tuning allows for adding new capabilities, improving performance in specific languages or domains, and creating models that can be hosted on private infrastructure.
• Tools for fine-tuning include Keras, Hugging Face Transformers, Onseloth, and Axelottle.
• Over 70,000 Gemma-based models have been built by the community.

Case Studies and Community Variants

• AI Singapore's CIO model: Fine-tuned Gemma 2 for Southeast Asian languages.
• Insight's Bulgarian LLM: Fine-tuned Gemma 2 to be the best Bulgarian LLM.
• Dolphin Gemma: Large language model for dolphin communication (coming soon).
• Mad Gemma: Gemma variant for the health industry, fine-tuned for medical use cases (available on Hugging Face and Vertex).
• Sign Gemma: Expanding Gemma to sign language, translating sign language to text (coming soon).
• Shujma: Safety classifier for images, based on Gemma 3 4B.

Conclusion

Gemma is a powerful and versatile open model family that is rapidly evolving thanks to community feedback and Google's ongoing development efforts. Its accessibility, multimodal capabilities, multilingual support, and focus on responsible AI make it a valuable tool for developers, researchers, and organizations looking to build innovative AI applications. The release of Gemma 3N and specialized variants like Mad Gemma and Sign Gemma demonstrate Google's commitment to expanding the reach and impact of open models.