Building a frontier AI search experience

Key Concepts

• Frontier AI Product
• AI Overviews
• AI Mode
• Query Fan-Out
• Model Customization
• Multimodal Search (Image, Voice, Video)
• Personalization
• Agentic AI
• Deep Search

Google Search as a Frontier AI Product

• Transition: Google Search is transitioning into a frontier AI product, leveraging advanced AI models to provide high-quality information and access to the world.
• Goal: To enable billions of users to ask anything of search and receive helpful, high-quality information.
• Components:
  • A model that understands the web and Google's information.
  • Experiences that make information easy to understand.
  • Unprecedented scale of operation.
• Example: Asking "How do I remove a ketchup stain from my jeans without ruining them?" triggers an AI overview with helpful content and links.
• Scale: Google Search is the largest AI product globally, with 1.5 billion monthly users of AI experiences (AI overviews, multimodal search).

AI Overviews and AI Mode

• AI Overviews: Provide helpful content and links in response to natural language questions.
• AI Mode: A frontier search experience that allows users to ask anything in Google Search.
  • Uses a state-of-the-art model to make plans, think, and perform Googling on the user's behalf.
  • Generates a "fan-out" of Google search queries to research and find information.
  • Available in the US and India.
• Query Fan-Out: The process of generating multiple Google search queries based on a user's question to gather comprehensive information.
  • Example: For "things to do in Nashville with a group," the system might generate queries for restaurants, bars, and activities for kids.

Model Customization and Deployment

• Balance: Finding the right model size for the question's complexity is crucial. Simple questions require instantaneous responses, while complex questions can justify longer processing times.
• Model Customization: A mix of custom models and general model improvements is used. Customization addresses unique needs, while rebasing to new models provides quality and helpfulness gains.
• Rapid Deployment: Frontier models are deployed at search scale quickly, requiring collaboration between search, infrastructure, and Google DeepMind teams.
• Gemini 1.5 Pro: Integration of Gemini 1.5 Pro has significantly improved search capabilities.

Deep Search

• Functionality: Deep Search can think, make plans, and perform hundreds of queries to find information.
• Use Case: Complex purchases or research tasks where users are willing to wait for a comprehensive response.
• Example: Researching a safe purchase, considering fire safety ratings, burglar protection, and insurance implications.
• User Notification: Users are notified when Deep Search has finished its research and can access the report in their history.

Multimodal Search

• Visual Search: The fastest-growing part of search, up 70% year-over-year, especially popular with younger users.
• Use Cases:
  • Shopping: Identifying and purchasing outfits seen on social media.
  • Homework: Getting help with visual problems.
  • Identifying objects: Flowers, buildings, plaques, books.
• Technology: Combines state-of-the-art models with Google Search's visual intelligence and image recognition capabilities.
• Visual AI Mode: Allows users to ask questions about images and receive visual replies.
  • Example: Asking "What's a good outfit to wear for a rehearsal dinner in Southern California near a beach?" and receiving a visual response with shoppable items.
• Speech: The goal is to enable natural conversations with Google Search, including live voice interactions.
• Search Live: An experiment in labs that allows users to have a natural conversation with search, leveraging the same models used for query fan-out and real-time information access.

Personalization

• Importance: Personalization is crucial for making search more efficient and relevant.
• Current Implementations: Recently visited labels, personalized search suggestions.
• Future Enhancements: Connecting Gmail to provide personal context for AI experiences.
• Opt-In: Personalization will be opt-in, respecting user privacy.
• Use Cases: Shopping recommendations based on past purchases, restaurant suggestions based on personal tastes.
• Principles: Personalization will be applied thoughtfully, avoiding bias in factual queries.

Agentic AI and Task Completion

• Trend: Enabling AI to perform tasks on the user's behalf.
• Project Mariner: Building an agentic version of search based on Gemini infrastructure.
• Example: An AI agent finding the perfect concert seats based on user preferences.
• Focus: Automating repetitive tasks and providing relevant information for decision-making.
• Availability: Checking availability (e.g., restaurant reservations, hotel rooms) is a key area for agentic AI.
• User Control: Users will retain control over final decisions and actions.

The Value of Search in the AI Era

• Increased Importance: As AI tools make it easier to create content, the value of search for finding authoritative and expert sources increases.
• Web Expansion: The web is expanding with more content, creating new opportunities for connection and discovery.

Bespoke Experiences and Software Creation

• Potential: Building bespoke experiences and software on the fly to answer user queries.
• Example: An AI model conducting market research, launching surveys, and analyzing data to answer a niche question.
• Travel Simulation: Building bespoke simulations to play out travel scenarios and provide personalized recommendations.

Collaboration and Teamwork

• Cross-Team Collaboration: Collaboration between Google Search, Google DeepMind, and infrastructure teams is crucial for building AI-powered search experiences.
• Strike Teams: Small, scrappy teams are formed to tackle specific projects and drive innovation.

Roadmap for AI in Search (Next 6 Months)

• Multimodal: Adding video input to search, allowing users to ask questions about what they are seeing live.
• Visualization and Richer Experiences: Enhancing inspirational questions with visual and interactive elements.
• Personalization: Expanding personalization based on user context, starting with Gmail integration.
• Agentic AI: Enabling AI to help users complete tasks, such as booking tickets and planning events.

Synthesis/Conclusion

Google Search is undergoing a significant transformation into a frontier AI product, leveraging advanced AI models like Gemini to provide more helpful, personalized, and efficient search experiences. This involves integrating multimodal search capabilities, enabling agentic AI for task completion, and customizing models to better understand user needs. The focus is on making information more accessible and actionable, ultimately empowering users to be more productive, efficient, and informed. The future of search involves a seamless integration of AI across various modalities and tasks, with a strong emphasis on personalization and user control.