PaliGemma – Making Gemma 2 see by adding a vision encoder

PaliGemma: Extending Gemma with Multimodality - Summary

Key Concepts:

• PaliGemma: Multimodal extension of Gemma models (Gemma & Gemma 2).
• Zero-shot object detection and segmentation.
• SigLIP: Vision encoder used to convert images to tokens.
• LLM: Large Language Model (Gemma).
• Vocabulary extension: Adding special tokens for bounding boxes and segmentation masks.
• Co-training: Jointly training the vision encoder and LLM.
• Mix Model: A pre-trained model provided for demonstration purposes.
• Fine-tuning: Crucial step for adapting PaliGemma to specific tasks.
• ColPali: Document retrieval system using PaliGemma.
• Pi Zero: Open-source robotics model based on PaliGemma.
• Gemma 3: The latest multimodal model with improved capabilities.
• Pan and Scan: Technique to increase resolution while maintaining aspect ratio.
• Instruction tuning: Training the model to follow instructions directly.

1. Introduction to PaliGemma

• PaliGemma extends the capabilities of previous Gemma versions (Gemma and Gemma 2) by adding multimodality.
• The presentation will cover what PaliGemma can do, how it's implemented, and when to use PaliGemma 2 versus Gemma 3.

2. Capabilities of PaliGemma

• General VLM Capabilities:
  • Extracting specific information from images.
  • Reading text from images (OCR).
  • Describing images in detail.
• Advanced Capabilities:
  • Zero-shot object detection: Identifying and localizing objects in images without prior training, providing bounding boxes. Example: Detecting puffins in an image.
  • Zero-shot segmentation: Creating pixel-level masks for different objects in an image without prior training. Example: Segmenting multiple objects with individual masks.

3. Implementation Details

• Architecture:
  • SigLIP Vision Encoder: Processes the input image and converts it into tokens.
  • Gemma LLM: Receives the image tokens and a prefix (instruction or question) as input.
  • Autoregressive Generation: Gemma generates an answer based on the input.
• Model Sizes and Resolutions:
  • PaliGemma comes in three different sizes, both in terms of parameters and input resolution.
  • Scaling input resolution has a similar effect on compute as scaling the model size.
  • The range of compute spans more than 155x, from the smallest model and resolution to the largest.
• Segmentation and Detection:
  • The Gemma vocabulary is extended with special tokens for bounding boxes and discrete tokens for segmentation masks.
  • A small pre-trained model upscales a 4x4 grid of discrete tokens to a continuous 64x64 mask within the bounding box.

4. Training Process

• Co-training: The pre-trained vision encoder and Gemma 2 LLM are co-trained to adapt to each other and learn detection and segmentation.
• Long Pre-training Phase: 1 billion examples of image-text data augmented with OCR annotations and pseudo-annotations for detection and segmentation from teacher models.
• Upscaling Phase: The model is upscaled to high resolution with a similar mix of data, augmented with more OCR data.
• Fine-tuning: Developers need to fine-tune the pre-trained models on their own data. The tech report provides guidance on hyperparameter selection.
• Mix Model: A model provided for demonstration purposes, but fine-tuning is crucial for optimal performance.

5. Performance and Benchmarks

• Academic benchmarks are used to evaluate the impact of model size and input resolution on performance.
• Some tasks benefit more from larger images (e.g., ChartQA augmented), while others benefit more from larger models (e.g., outside knowledge VQA).
• Fine-tuning on specific data leads to state-of-the-art results in various applications.
  • Table structure recognition.
  • Chest X-ray report generation.
  • Molecular structure recognition.
  • Optical music score recognition.

6. Real-World Applications

• ColPali: A document retrieval system that uses PaliGemma to embed PDF screenshots and retrieve documents based on query embeddings. This approach is simpler and achieves state-of-the-art results compared to previous methods.
• Pi Zero: An open-source robotics model based on PaliGemma. It takes input images and instructions, and outputs action tokens to control robot effectors. It can be used for tasks like grinding coffee or folding laundry.

7. Gemma 3 vs. PaliGemma 2

| Feature | PaliGemma 2 | Gemma 3