Gemma 3: NEW Opensource Multimodal Model Beats DeepSeek V3 & o3 Mini! (Fully Tested)

Key Concepts

• Gemma 3: Google's new collection of lightweight, state-of-the-art open AI models.
• Parameter Sizes: 1B, 4B, 12B, and 27B parameter models.
• Multimodal Infusion: Ability to support text, images, and short videos (excluding the 1B model).
• Token Support: Up to 128k tokens (32k for the 1B model).
• Open Source: Models can be installed locally using tools like Llama or LM Studio.
• AI Studio: Google's platform for interacting with Gemma 3 models for free.
• Benchmarks: Performance comparisons against models like DeepSeek V3 and Llama 3.
• Hugging Face: Platform for accessing Gemma 3 model endpoints.
• AMA: Tool for local installation of Gemma 3.
• SVG Code Generation: Generating Scalable Vector Graphics code.

Gemma 3 Overview

Google has released Gemma 3, a new family of open AI models designed for efficiency and performance on a single GPU or TPU. Gemma 3 is built on the same technology as Google's Gemini 2.0. The models come in four sizes: 1B, 4B, 12B, and 27B parameters, optimized for devices ranging from phones to workstations. These models are pre-trained in 140+ languages and natively support 35+.

Model Capabilities

Excluding the 1B model, Gemma 3 models are multimodal, supporting text, images, and short videos. The models support up to 128k tokens, except for the 1B model, which supports 32k. Gemma 3 outperforms larger models like DeepSeek V3 (67B parameters) and Llama 3 (405B). It even outperforms 03 Mini. Gemma 3 requires only one Nvidia H100 GPU, while competing models require multiple. Compared to Gemma 2, Gemma 3 shows significant improvements across benchmarks like math, coding, general Q&A, and logical reasoning.

Installation and Access

Gemma 3 can be deployed on phones, web, or cloud using Google's AI Dodev website. Models can be accessed via Hugging Face or installed locally using AMA. In LM Studio, users can search for Gemma 3 models, select a GGF version, and download the desired quantization size. Users can also interact with Gemma 3 models for free on Google's AI Studio by selecting the desired model size (e.g., 27B parameter model).

Benchmark Testing

The video assesses the Gemma 3 27B parameter model using eight prompts across different categories:

1. Web App Generation: The model was tasked with building a web app for logging monthly expenses and income with data visualization using HTML, CSS, and JavaScript. The model generated a detailed app with transaction logging, financial summaries, and transaction history, which was considered impressive.
2. Image Understanding: The model was asked to create a short story based on a series of images, including a dog with a croissant. The model demonstrated good object recognition and scene understanding, generating a relevant story.
3. SVG Code Generation: The model was asked to generate SVG code for a symmetrical butterfly. The generated code failed to produce a recognizable butterfly, indicating a weakness in this area.
4. Mathematics: The model was given a simple algebra equation to solve for x. The model correctly solved the equation, demonstrating strong math capabilities. The answer was X = 3 or 1.
5. Logical Reasoning: The model was presented with a problem involving a farmer, cows, goats, and chickens to calculate total milk production in a week. The model correctly used logic and deduction to arrive at the answer of 885 liters.
6. Debugging and Error Analysis: The model was given a Python function with a bug that incorrectly added odd numbers to a sum of even numbers. The model correctly identified and fixed the bug.
7. Common Sense Reasoning: The model was asked what would happen if a bowl of water was placed outside in freezing temperatures and to explain why. The model accurately described the freezing process, including the science behind it (temperature, molecular motion, freezing point, heat transfer, latent heat of fusion).

Overall Assessment

Gemma 3 is a great model for its size, though the prompts used were not extremely complex. It competes well against larger models like DeepSeek V3. While it has some weaknesses in coding (specifically SVG generation), it excels in math, multimodal tasks, and general knowledge. It is a strong, accessible, and open-source model suitable for deployment on various devices.

Conclusion

Gemma 3 is a powerful and accessible open AI model family from Google, offering a balance of performance and efficiency. Its multimodal capabilities and strong performance in math and reasoning make it a valuable tool for a wide range of applications. While it has some limitations in specific coding tasks, its overall capabilities are impressive for its size and accessibility.