Why are weather forecasts so hard to get right? | Microsoft Azure and NVIDIA | Catalyst E4

Weather Prediction, AI, and Data: A Deep Dive into Tomorrow.io's Approach

Key Concepts:

• Nowcasting: Very short-range weather forecasting (0-6 hours) focusing on current conditions.
• Operational Weather Prediction: Traditional weather forecasting methods used for decades, relying on initial conditions updated every six hours.
• Intrinsic Predictability: The inherent limits of weather prediction due to chaotic systems (the “butterfly effect”).
• Practical Predictability: Limitations in weather prediction due to computational power and data availability.
• CorrDiff: NVIDIA’s corrective diffusion model, a novel approach to improving grid resolution in weather forecasting.
• FOCUS Model: Tomorrow.io’s AI model built on Earth-2 and PhysicsNeMo from NVIDIA, tailored for specific use cases.
• Earth-2: NVIDIA’s platform for physics-based models of the atmosphere.
• PhysicsNeMo: NVIDIA’s framework for developing and deploying physics-informed machine learning models.
• Containerized Applications: Software packaged with all its dependencies, allowing for scalable and portable deployment.
• Microwave Sounders: Instruments used on Tomorrow.io’s satellites to measure atmospheric temperature and humidity.
• 6U CubeSat: A small satellite format, approximately the size of a loaf of bread.

The Scale and Challenges of Weather Data

The video emphasizes the pervasive impact of weather on all aspects of life, highlighting that its data is immense and complex. The sheer dimensionality of Earth’s weather systems presents significant challenges for atmospheric scientists. Traditional weather forecasting, reliant on initial conditions released every six hours, struggles to capture rapidly evolving events. A critical limitation is the lack of real-time observation coverage, particularly over oceans where many weather systems originate. This delay hinders timely action and accurate prediction. The speaker illustrates this complexity by pondering the weather patterns that create seemingly simple phenomena like freshly fallen snow, questioning the radiative balance, moisture sources, and dynamic drivers.

The Rise of AI in Weather Prediction

While machine learning is rapidly advancing, improvements in weather prediction are “hard fought.” Historically, AI-based models only recently (within the last 2-4 years) began to surpass the skill of traditional operational weather prediction models that have been in use for 70+ years. This underscores the difficulty of accurately modeling atmospheric processes. The speaker notes that “any gains, any improvements are really hard fought.”

Tomorrow.io’s Multi-Model Approach & Resilience Platform

Tomorrow.io focuses on distributing weather insights and data to a global customer base through its Resilience platform and API. This platform allows customers to set up tailored insights driving their business operations. The core of Tomorrow.io’s approach is a “multi-model” strategy, combining internally developed AI models with those from other sources. This acknowledges that no single model will consistently outperform others in the long term, emphasizing the importance of continuous community improvement. Operationalizing these models – applying them to specific business outcomes – and leveraging the best available data are crucial.

The platform relies on a scalable infrastructure utilizing containerized applications to run various models (nowcasting, forecasting) and handle diverse data sources (radar, satellite). Tomorrow.io leverages Azure and NVIDIA GPUs for processing power.

Leveraging NVIDIA’s Technology: FOCUS, Earth-2, and CorrDiff

A key partnership with NVIDIA has driven research and development in weather modeling. Tomorrow.io’s FOCUS model is built upon open-source code from NVIDIA’s Earth-2 and PhysicsNeMo. This allows them to address fundamental limitations in predictive accuracy for their customers.

• Earth-2 provides access to physics-based models of the atmosphere, enabling more accurate representation of Earth’s systems.
• PhysicsNeMo is a framework for developing physics-informed machine learning models.
• CorrDiff (Corrective Diffusion), a recent NVIDIA innovation, addresses challenges with grid resolution. It learns to emulate complex physical processes from vast datasets, improving the representation of phenomena like thunderstorm development and evaporative cooling. CorrDiff’s ability to capture these interactions allows for more flexible model deployment on Azure, tailored to specific client needs.

The combination of these technologies allows Tomorrow.io’s scientists to “chase the problem” and improve predictions of severe weather events, expanding the probabilities of accurate forecasts and enhancing risk management for customers.

Addressing Data Gaps with Satellite Constellations

Recognizing the “Holy Grail” is the lack of sufficient data, Tomorrow.io initially a software-only business, embarked on developing its own satellite constellation. Traditional satellite development takes 10-15 years, but Tomorrow.io is adopting a faster, “tech refresh” approach, developing satellites with shorter lifespans (2-3 years) to leverage rapid technological advancements.

Their satellites are unique due to their wide swath width and compact size (6U CubeSat – about the size of a loaf of bread). The current constellation is “hybrid,” consisting of two radars and seven microwave sounders (expanding to 18), generating data that feeds into their models. The company believes that more data, particularly high-resolution data, leads to improved predictions, and AI is helping to break the traditional paradigm of diminishing returns. Their goal is to achieve a sub-hourly revisit rate for global weather data.

Modernizing Meteorological Infrastructure & Democratizing Access

Microsoft and NVIDIA are collaborating to develop technologies that accelerate AI’s impact on the weather industry. Together with Tomorrow.io, they offer solutions to modernize meteorological infrastructure, even in countries with unreliable systems. This allows for rapid improvements in forecasting capabilities.

Bridging the Gap Between Atmospheric Science and Machine Learning

The speaker, an atmospheric scientist, acknowledges the challenges of integrating new machine learning tools into traditional scientific workflows. The ability to modify containerized applications and iterate through experiments on Azure, leveraging NVIDIA GPUs, allows them to deliver better weather insights without needing to develop every component from scratch.

The Impact and Future of Weather Prediction

Ultimately, Tomorrow.io’s work is driven by a desire to save lives, reduce accidents, and improve business outcomes. The speaker emphasizes the privilege of working on something that benefits every citizen globally. The company’s focus on data acquisition, advanced modeling, and scalable infrastructure positions them to continue pushing the boundaries of weather prediction and its applications.