Building with Gemini Embedding 2: Our first natively multimodal embedding model

Key Concepts

• Gemini Embedding 2: Google DeepMind’s first natively multimodal embedding model.
• Unified Embedding Space: A single vector space where text, images, video, audio, and documents are mapped together.
• Matryoshka Representation Learning: A technique that nests critical semantic information in the earliest dimensions of a vector, allowing for flexible output sizes.
• Multimodal RAG (Retrieval-Augmented Generation): An architecture where the model serves as the retrieval backbone for agents to query diverse media types simultaneously.
• Cosine Similarity: The mathematical method used to measure the distance between vectors to determine semantic relevance.

---

1. Overview of Gemini Embedding 2

Gemini Embedding 2 is designed to process multiple modalities (text, image, video, audio, documents) natively. By avoiding intermediate text conversions, the model maintains semantic relationships across different media types.

• Interleaved Inputs: Developers can pass multiple modalities (e.g., an image and a text description) in a single API request to generate one composite embedding.
• Multilingual Support: The model supports over 100 languages out of the box.
• Performance: It establishes a new standard for multimodal depth, outperforming leading models in text, image, and video tasks while providing strong speech capabilities.

2. Flexible Output Size and Optimization

To balance storage costs and search latency, the model utilizes Matryoshka Representation Learning.

• Dimensionality Options: Users can choose between 3072 (default/maximum precision), 1536, or 768 dimensions.
• Quality Maintenance: The model is designed to retain high semantic quality even when truncated to lower dimensions, allowing developers to tune the cost-performance trade-off.

3. Implementation and Frameworks

The model is accessible via the Google AI Python SDK. The workflow generally follows these steps:

1. Initialization: Set up the client and specify the model ID (gemini-embedding-2).
2. Ingestion: Read files (images, audio, PDFs, video) as bytes and define the appropriate MIME type.
3. Embedding Generation: Use client.models.embed_content to generate vectors.
4. Aggregation: Multiple inputs (e.g., a text description + an image + an audio file) can be appended to a single contents list to produce one unified embedding.
5. Configuration: To adjust dimensionality, pass a dictionary with the output_dimensionality key in the request configuration.

4. Multimodal Search Methodology

The video demonstrates a practical application of the model for similarity search:

• Indexing: Iterate through a dataset, generate embeddings for each item, and store them (e.g., in a JSON file) alongside their metadata.
• Querying: Generate an embedding for a query (which can be text, an image, or audio).
• Similarity Calculation: Use Cosine Similarity to compare the query vector against the stored database vectors.
  • Formula: $\text{Similarity} = \frac{A \cdot B}{|A| |B|}$
• Cross-Modal Retrieval: The model allows for "image-to-image," "text-to-image," and "audio-to-text" retrieval, enabling users to find a "cat purring" audio file by searching with the text "cat."

5. Use Cases and Applications

• Multimodal RAG Agents: Building agents that query video libraries, meeting audio, and text documents simultaneously without needing separate ingestion pipelines.
• Task-Specific Optimizations: The model is optimized for:
  • Search queries and question answering.
  • Fact-checking and code retrieval.
  • Classification, clustering, and semantic similarity.

6. Synthesis and Conclusion

Gemini Embedding 2 simplifies the developer experience by providing a single, unified endpoint for all media types. By leveraging Matryoshka representation learning, it offers a scalable solution for high-performance retrieval systems. The ability to perform cross-modal searches—such as finding images based on audio descriptions—positions this model as a powerful tool for modern, agentic AI workflows. Developers are encouraged to use the provided Google AI Studio app and documentation to implement these capabilities in their own applications.