Google AI Infrastructure PM On New TPUs, Liquid Cooling and More

Key Concepts:

• AI Infrastructure (GPUs, TPUs)
• TPU (Tensor Processing Unit) generations (Ironwood, Trillium, V5P, V5E)
• Liquid Cooling
• Inter-chip Interconnect
• Pathways (distributed computing framework)
• Jax
• Very Large Model (VLM)
• Inference vs. Training
• Performance per Watt
• Open Compute Project
• Green Concrete
• Data Center Sustainability
• GB200, B200, A4, A4X (Nvidia GPUs and Google Cloud VMs)

AI Infrastructure and TPU Development

• Chelsea Chop, Senior Product Manager for AI Infrastructure at Google Cloud, discusses the challenges and advancements in AI infrastructure, focusing on TPUs and GPUs.
• AI infrastructure encompasses GPUs and TPUs, requiring a feedback loop from customers to ensure product design meets their needs for efficiency and usability.
• The latest generation of TPUs, Ironwood, features significant power improvements, achieved through incremental design changes and a focus on power and thermal constraints.
• Liquid cooling is a critical component, with Google on its fourth generation of deployment since 2017. Liquid cooling provides incredible efficiency in cooling chips at scale.
• Liquid cooling implementation involves visible external pipes for leak detection, demonstrating continuous evolution and lessons learned.

TPU Architecture and Networking

• The largest TPU machines consist of over 9,000 chips (specifically, 9,260).
• Inter-chip interconnects, relying on light reflection, enable scaling and connectivity between pods.
• Google is continuously exploring new networking technologies to enhance TPU performance.

Hardware-Software Co-design and Pathways

• A tight relationship between hardware and software teams is crucial for efficient resource utilization and easy application deployment.
• Google integrates AI research and innovation from DeepMind and core Google into Google Cloud, making it accessible to users.
• Pathways, a distributed computing framework published in 2017, facilitates scalable training and inference.
• Pathways is now available for use with TPUs and Jax on Google Cloud, enabling users to leverage Google's research in foundational models like Gemini.

Model Development and Hardware Cadence

• The rapid pace of model architecture development presents a challenge for hardware development, which operates on a slower cadence (TPUs on roughly a one-year cadence).
• Software optimization plays a key role in maximizing the utilization of available hardware.
• Google's design philosophy emphasizes software resilience to mitigate hardware failures, contrasting with IBM's focus on hardware redundancy.
• TPU design considers future model architectures, even with uncertainty about their exact form.
• Current focus includes "think time compute," integrating inference and training.

GPU and TPU Considerations

• Nvidia is recognized as an important partner with a strong ecosystem.
• Google was the first cloud provider to bring VMs for B200 and GB200 (A4 and A4X) to market.
• The choice between GPUs and TPUs depends on the specific workload, use case, and existing team expertise.
• A "better together" approach is often viable, leveraging the strengths of both GPUs and TPUs.
• Education and guidance are provided to customers to help them make informed decisions about hardware selection.
• Very Large Model (VLM) is useful for easily porting and testing workloads across different hardware.

Data Centers and Sustainability

• Data center visits are encouraged to understand the scale and innovations involved.
• Google's data center innovations, including green concrete and AI-powered robotics, are contributed to the Open Compute Project.
• Sustainability is a key consideration in data center design and operation.
• TPUs were initially developed to address the compute demands of voice search, highlighting the need for energy-efficient AI-specific hardware.
• Performance per watt is a key metric for evaluating TPU efficiency.

Liquid Cooling Implementation

• Not all systems are liquid-cooled; the decision depends on design tradeoffs.
• Ironwood and TPU V5P are liquid-cooled, while Trillium and TPU V5E are not.
• Each cooling unit can serve an entire pod, demonstrating the scalability of the liquid cooling system.
• The liquid used is ambient temperature water.
• Google's custom liquid cooling is a differentiator for its A4X offering of GB200, enabling higher performance by maintaining optimal temperatures.

Conclusion:

The discussion highlights Google's advancements in AI infrastructure, particularly with TPUs and liquid cooling technologies. The importance of hardware-software co-design, the challenges of keeping pace with rapid model development, and the focus on sustainability in data center operations are emphasized. The interplay between GPUs and TPUs is presented as a collaborative ecosystem, with the optimal choice depending on specific use cases and workload requirements.