Google's AI stack for developers

Google's AI Stack for Developers: Detailed Summary

Key Concepts:

• Google's AI Stack: End-to-end ecosystem for AI development.
• Foundation Models: Gemini, Gemma, domain-specific models.
• AI Frameworks: Jax (research), Keras (applied AI), PyTorch.
• Developer Tools: Google AI Studio, Gemini API, GenAI SDK.
• Infrastructure: TPUs, XLA (machine learning compiler), VLM (inference engine).
• Google AI Edge: Framework for on-device ML deployment.
• Alpha Evolve: Gemini-powered coding agent.
• AI Co-scientists: AI agent for scientific discovery.
• Gemini Robotics: Models for controlling robots.

1. Introduction and Overview

• Ron Kashka and Josh introduce Google's AI stack for developers, emphasizing Google's decades-long leadership in AI.
• Mission: Empower every developer and organization to harness the power of AI.
• Google's AI stack combines robust infrastructure with state-of-the-art research, enabling real-world applications.
• The talk covers foundation models, AI frameworks, developer tools, infrastructure, and on-device deployment.

2. Foundation Models: Gemini

• Gemini models are the most capable and versatile model family, known for being multimodal, having a long context window, and powerful reasoning.
• Gemini 2.5 Pro: Most advanced model, excels at complex tasks, deep reasoning, and coding. Leads coding benchmarks like WebDev Arena.
• Gemini 2.5 Flash: Efficient and fast, improved across reasoning, coding, multimodality, and long context.
• Gemini 2.0 Flash: Fast and cheap, suitable for general-purpose tasks.
• Gemini Nano: Optimized for on-device tasks.

3. Google AI Studio and Gemini API

• Google AI Studio: Simplest way to test the latest models, prototype, and experiment without Google Cloud knowledge. Free of charge.
• New "Build" tab in AI Studio instantly generates web apps from natural language.
• New generative media experience in AI Studio.
• Built-in new dashboard based on community requests.
• Native audio and TTS support in AI Studio.
• Gemini API: New capabilities for text-to-speech, allowing control over emotion and style. Use cases include dynamic audiobooks, engaging podcasts, and natural customer support voices.
• Enhanced tooling: Routing with Google Search and code execution in one API call.
• URL context: Provides the model with depth content from web pages, enabling powerful search agents.
• Gemini SDK support for MCP reduces developer friction and simplifies building agent capabilities.
• Example: Mumble Jumble app in AI Studio allows natural language interaction for dynamic audio experiences. Demonstrates 2.5 preview native audio dialogue with customizable voices and languages.
• Example: Prompting the model to roll a dice twice and calculate the probability of the result being seven, showcasing "thought summaries" that explain the model's reasoning.

4. Gemini Developer API

• Easiest way to develop with Google's foundation models.
• Resources: ai.google.dev (developer documentation), Gemini API cookbook (go.gle/cookbook) with end-to-end examples.
• Easy setup: Get an API key in Google AI Studio without a credit card.
• GenAI SDK: User-friendly SDK for the Gemini API.
• Advanced functionality: Access thinking summaries with a single line of code (e.g., thinking_config=types.ThinkingConfig(include_thoughts=True)).
• Function calling: Pass function definitions to the Gemini API in JSON, allowing the model to determine whether to call the function based on the prompt. Enables building agents that can interact with external tools and services.
• Example: Passing a Python function definition for getting the weather to the Gemini API, allowing the model to determine when to call the function based on the user's prompt (e.g., "What's the temperature in London?").

5. Generative Media Models

• Powerful suite of generative media models for transforming creative experiences across images, video, and audio.
• New generative media console in AI Studio allows creating and interacting with creative models.
• Access to image generation, video generation, and music generation models with applets to get started.
• Lyria: Music generation model powering music FXDJ. Available in the API and AI Studio.
• Allows everyone to interact to create and to perform generative music in real time.

6. Gemma

• Gemma 3: Most advanced model, comes in four sizes (1, 4, 12, and 27B).
• Offers developers the flexibility to optimize performance for diverse applications.
• 42 and 27B are multimodal, multilingual, and have a long context window (up to 128,000 tokens).
• Med Gemma: Most capable collection of open models for multimodal medical text and image comprehension. Available in 4B and 27B.
• Gemma 3N: Optimized for on-device operation on phones, tablets, and laptops.
• Gemmaverse: Booming with new variants being developed.
• One-click deployment of Gemma models from AI Studio to Cloud Run.

7. AI Frameworks: Keras and Jax

• Keras: Easiest way to fine-tune a model for applied AI.
• Requires a two-column CSV file (prompt and response).
• Tutorial: Import a Gemma model from Keroshub, prompt it, and do Laura fine-tuning in a few lines of code.
• Jax: Python machine learning library for research and large-scale deployments.
• Scales easily to tens of thousands of accelerators.
• At its core, Jax is a Python machine learning library with a NumPy API.
• Transforms: grad (gradients), JIT (JIT compilation).
• Ecosystem of libraries for optimizers, checkpoints, and neural networks.
• Max Text and Max Diffusion: GitHub libraries with reference implementations of large language models and diffusion models.
• Marin: Fully open model released by Stanford University, built with Jax and TPUs. Shares weights, architecture, data sets, and code.
• Tunix: New library for tuning in Jax, being built with the community.

8. Infrastructure: XLA and VLM

• XLA: Compiler for machine learning code, used by Jax, Keras, TensorFlow, and PyTorch.
• Takes Python code, performs optimizations, and prepares it to run on accelerators.
• Portable: Can run on TPUs, GPUs, and other accelerators.
• PyTorch now works with XLA.
• VLM: Super popular inference engine.
• TPU support added for VLM, available to PyTorch developers.
• Jax support being added to VLM.
• LLMD: New partnership between Red Hat, Nvidia, and Google for distributed serving, aiming to bring the best of serving into open source and make it available to everybody.

9. Google AI Edge

• Framework for deploying machine learning models on Android, iOS, browsers, and embedded devices.
• Reasons for deploying on mobile: latency, privacy, offline access, cost savings.
• Support for the latest Gemma models.
• Hugging Face community contributing pre-optimized models for on-device deployment.
• AI Edge portal (private preview): Testing service to verify model performance on a fleet of real devices.

10. Future Directions and Applications

• Pushing the boundaries of what's possible with AI.
• Areas with huge potential: scientific discovery, healthcare.
• Alpha Evolve: Gemini-powered coding agent for designing advanced algorithms.
• AI Co-scientists: AI agent for accelerating scientific discovery and drug development. Uses a coalition of different agents inspired by the scientific method.
• Gemini Robotics: Advanced vision language action models for controlling robots. Robot agnostic and uses multi-embodiment.

11. Conclusion and Call to Action

• Encourages developers to engage with Google's AI stack, provide feedback, and co-create the future of AI.
• Provides links to resources: ai.google.dev, Google AI Studio, Jax and Keras documentation, openXLA.org, Google AI Edge.
• Highlights upcoming sessions on the Gemini API, Gemmaverse, and robotics.