Under the hood with Google AI

Key Concepts

• Natively Multimodal Models: Models designed from the outset to process various data types (text, images, audio, video, actions).
• Bespoke Models: Specialized models created for specific tasks or domains, independent of general-purpose models.
• Query Fan Out: A technique used in search where the model breaks down a complex query into multiple sub-queries to gather information from various sources.
• Personal, Proactive, and Powerful (3 Ps): The guiding principles for developing a universal AI assistant.
• AI Overviews: AI-generated summaries and insights provided in search results.
• Deep Search: A more advanced search feature that takes on longer-term, complex tasks.
• Tool Calls/Function Calls: The ability of AI models to use external tools and functions to enhance their capabilities.
• Action API: An API that allows models to perform UI actions on a computer.
• Content Studio API: An internal API that turns a bag of content into an interesting show.
• Evals (Evaluations): Metrics and benchmarks used to assess the performance and capabilities of AI models.

Gemini and Multimodality

• Core Argument: Building AGI requires natively multimodal models that can process various perceptive streams, including video, audio, text, and actions.
• Details:
  • Multimodality extends beyond video, audio, and text to include actions, tool calls, and function calls.
  • The Gemini program aims to integrate diverse capabilities (coding, text generation, image generation) into a single model to leverage transfer learning effects.
  • Example: Wipe coding, where a model understands the world and can generate code from high-level descriptions and sketches.
• Bespoke vs. Mainline Gemini:
  • Bespoke models are essential for exploring specific ideas and proving concepts in a controlled environment.
  • The Gemini program focuses on rapidly integrating successful ideas from bespoke models into the mainline Gemini model.
• Diffusion Gemini Experiment:
  • A research experiment exploring text generation using diffusion models, which have advantages in complex domains like coding.
  • Diffusion models allow for correcting mistakes by revisiting the output space.

Scaling AI Models

• Key Factors: Model performance depends on various factors, including model size, data, model architecture, and inference time.
• Data:
  • Data is crucial, encompassing data acquisition, representation, and understanding correlations between different data streams (especially in multimodality).
• Model Architecture:
  • Model architecture, algorithms, and optimization methods are essential for improving model performance.
• Inference Time:
  • Techniques to improve inference time are multiplicative, enhancing the effectiveness of a model's capacity.
• Algorithmic Improvements:
  • Algorithmic improvements are critical and contribute significantly to overall progress.

Gemini's Impact on Search

• Challenge: Traditionally, search was limited to answering questions with information readily available in one place.
• Solution: Gemini enables the "query fan out" technique, where the model breaks down complex queries into sub-queries, gathers information from various sources, and synthesizes a comprehensive answer.
• Benefits:
  • Answering complex questions with multiple constraints.
  • Overcoming language barriers by translating content from different languages.
• AI Mode Rollout:
  • Prioritizes trustworthiness, accuracy, and helpfulness.
  • Considers latency as a critical factor, balancing response quality with speed.
  • Employs techniques like bolding to improve information skimming efficiency.
• Deep Search (Coming Soon):
  • Addresses longer-term, complex tasks where users are willing to wait longer for expert-level reports.
  • Provides users with options to choose the level of effort and time they want to invest in a task.

The Gemini App and Universal AI Assistant

• Vision: Building a truly universal AI assistant that is personal, proactive, and powerful.
• Three Ps:
  • Personal: Leveraging user data across Google to provide personalized assistance.
  • Proactive: Anticipating user needs and providing relevant information at the right moment.
  • Powerful: Utilizing advanced capabilities like video, imagery, and coding to assist users with complex tasks.
• Content Remixing:
  • Content is now infinitely remixable, allowing for the creation of new experiences from various data types.
  • Example: Transforming documents into podcasts or images/videos into multi-step action plans.

Tool Use and Agentic Models

• Tool Calls/Function Calls: Considered the next big frontier in AI, enabling models to use external tools and functions.
• Native Search Integration: Critical for Gemini, allowing the model to access fresh and factual information from the web.
• Action API: Enables models to perform UI actions on a computer.
• Future Vision: Models will become more like systems, with embedded functionalities and the ability to create and use their own tools.
• Model-as-a-System: The models are getting to a state where they do tool calls and function calls.

Google Labs and New Product Experiences

• Focus: Exploring workflows and the future of various industries to create innovative AI-powered products.
• Methodology:
  • Small teams (5-10 people) build and ship products quickly.
  • Experiments are launched on labs.google to gather user feedback.
  • Successful experiments are integrated into other Google products.
• Content Studio API:
  • Turns a bag of content into an interesting show.
  • Used in Search, Gemini App, Discover Feed, and Cloud.
• Key Insight: Identify user pain points and develop solutions that address them effectively.

Developer Opportunities and Advice

• Agility: Be agile and constantly play with the tech, adapting to the rapid pace of AI advancements.
• Persistence: Don't give up easily; ideas that don't work initially may become viable with newer model versions.
• Experimentation: Continuously try new things and adapt to changing user expectations.
• Evals:
  • Use academic evals, benchmarks, and product signals to assess model performance.
  • Balance quantitative metrics with qualitative insights and product vision.
  • Periodically reassess the effectiveness of evals and ensure they align with product goals.
• Embrace Non-Coders: Enable people who never thought they could write code to write lots of code.

Rapid Fire

• Future of Search: From information to intelligence.
• New Model Capability: Robotics.
• Biggest Developer Opportunity: People who never thought they could write code are going to write lots of code.

Synthesis/Conclusion

The discussion highlights Google's commitment to advancing AI through multimodal models, scaling techniques, and innovative product development. Gemini plays a central role in this vision, enabling new capabilities in search, AI assistance, and developer tools. The key takeaways emphasize the importance of agility, experimentation, and a user-centric approach to building successful AI products. The future of AI lies in creating intelligent systems that seamlessly integrate into people's lives, empowering them to achieve more with less effort.