ShieldGemma 2 – Developing safe and responsible AI for images

Key Concepts

• ShieldGemma 2: A 4 billion parameter image safety classifier built on Gemma 3.
• Image Safety Classification: Identifying whether an image violates predefined safety policies.
• In-Context Learning: Training a model by providing examples within the prompt.
• Policy Customization: Defining specific rules and guidelines for image safety.
• Input/Output Filter: Using ShieldGemma 2 to filter images before or after processing by other models.
• Harm Categories: Types of unsafe content, such as sexually explicit, violent, and dangerous content.
• F1 Score: A metric used to evaluate the performance of a classification model, balancing precision and recall.
• Prompt Engineering: Designing effective prompts to guide the model's behavior.
• Downstream Tasks: Applications that utilize the output of ShieldGemma 2, such as online filtering and model distillation.

Introduction to ShieldGemma 2

Dana Kurniawan and Wenjun Zeng introduce ShieldGemma 2, a 4 billion parameter image safety classifier built upon the Gemma 3 model. They emphasize that safety is a crucial investment in the open model ecosystem. ShieldGemma 2 is designed to be flexible and adaptable for various use cases.

How ShieldGemma 2 Works

ShieldGemma 2 takes a defined policy (customizable) and an input image. It outputs probabilities for "yes" or "no" tokens, indicating whether the image violates the policy. "Yes" signifies a policy violation, while "no" indicates compliance.

Use Cases

ShieldGemma 2 can be used as:

• Input Filter: To pre-screen images before they are processed by vision language models like Gemma 3.
• Output Filter: To filter images generated by image generation systems.

This dual-use purpose accommodates developer creativity and various downstream applications.

Step-by-Step Practical Application

1. Input Image: Start with an existing image.
2. Policy Customization: Customize the safety policy, focusing on key harms like sexually explicit, violent, and dangerous content. The prompt template allows for further customization.
3. Safety Policy Probabilities: ShieldGemma 2 outputs probabilities of tokens per image per policy.
4. Adaptation: Adapt the output for specific needs, such as online filtering, offline evaluation, model distillation, or downstream use cases.

The process is summarized as "image, policy, and label."

Performance Evaluation

Wenjun Zeng presents evaluation results based on internal datasets, comparing ShieldGemma 2 to models like LlavaGuard 7B, GPT-4o Mini, and Gemma 3 4B. The x-axis represents key harm types (sexually explicit, dangerous content, violence), and the y-axis represents the optimal F1 score. ShieldGemma 2 (represented by blue bars) outperforms other models across all key harm types.

Prompt Instruction Templates

The prompt template consists of:

1. Preamble: Defines the problem.
2. Policy Placeholder: An XML tag where the policy is specified.
3. Suffix: Instructs the model to provide a "yes" or "no" response indicating policy violations.

Python Code Implementation

The Python code demonstrates how to run ShieldGemma 2 and obtain results.

• Policy Customization: Users can define customized policies in a map, linking policy names to detailed definitions.
• Model Loading: The ShieldGemma for image classification class is used to load the model.
• Input Preparation: Images, default policies, and customized policies are prepared as inputs.
• Prediction: A one-click call to the model generates predictions.
• Output: Predicted model probabilities represent the likelihood of an image violating each defined policy. These scores can be used for downstream tasks like safety evaluation or distillation.

Conclusion

Dana Kurniawan concludes by emphasizing the importance of the open-source community and encourages users to try ShieldGemma 2, adapt it to their needs, and provide feedback. She acknowledges the team behind ShieldGemma 2 and expresses excitement for future improvements and evolution.