That time Google Cloud Platform bricked the Internet…

Google Cloud Outage: A Software Engineering Perspective

Key Concepts:

• Google Cloud Platform (GCP) outage
• Service Level Agreement (SLA)
• API Management Service
• Quota and Policy Checks
• Null Pointer Exception
• Rollback
• PostHog
• Max (PostHog's AI product analyst)

1. The Incident and its Impact:

• On June 12th, 2025, a faulty code deployment on Google Cloud Platform (GCP) caused a widespread internet outage.
• Services like Snapchat, Spotify, and Discord experienced disruptions.
• Cloudflare's Workers KV service suffered near 100% error rates for over two hours.
• Google's own services, including Gmail, Google Calendar, and Drive, were also affected.
• The outage resulted in significant financial losses for companies relying on GCP.

2. Service Level Agreement (SLA) and Financial Implications:

• GCP, like other cloud providers, offers a Service Level Agreement (SLA) guaranteeing a certain level of monthly uptime (typically 99.99% or greater).
• The outage violated the SLA, potentially entitling affected customers to financial compensation in the form of SLA credits (refunds).
• While the refunds might cost Google millions, the reputational damage is more significant, especially considering Google Cloud's third-place position in market share behind Azure and AWS.

3. Root Cause Analysis: The Faulty Code:

• The issue stemmed from a new quota policy check added on May 29th, 2025, to the API management service.
• This service is responsible for authorizing API requests and managing quota and policy information.
• The code path for the new feature was never properly executed during the staging phase because the policy change that would trigger the code was never pulled.
• The code lacked proper error handling, specifically a check for null pointers.

4. The Trigger and the Crash:

• On June 12th, a policy change was implemented and replicated globally.
• This triggered the previously dormant code path, leading to a null pointer exception.
• The null pointer caused the API management binary to crash, resulting in a crash loop and the widespread outage.

5. The Rollback and Recovery:

• Google developers had implemented a "big red button" for rolling back changes.
• The rollback process took approximately 40 minutes to initiate and four hours to fully stabilize the system.

6. PostHog and Max: Building Better Products:

• PostHog is presented as an all-in-one platform for building better products.
• Max, PostHog's AI-powered product analyst and assistant, is highlighted as a new feature.
• Max is deeply integrated with PostHog's data, enabling users to:
  • Research answers using natural language.
  • Generate data visualizations.
  • Get assistance within the PostHog UI.
  • Answer questions about documentation.
• PostHog offers analytics, feature flags, and session replays.

7. Technical Terms and Concepts:

• API Management Service: A service that handles API requests, authorization, and quota management.
• Quota and Policy Information: Data related to usage limits and access rules for APIs.
• Null Pointer Exception: An error that occurs when a program attempts to access a memory location through a null pointer (a pointer that doesn't point to a valid memory address).
• Crash Loop: A situation where a program repeatedly crashes and restarts.
• Rollback: Reverting a system or application to a previous, stable state.
• Staging Phase: A pre-production environment used for testing and validating code changes before deployment to production.

8. Synthesis/Conclusion:

The Google Cloud outage was a significant event caused by a combination of factors: a faulty code deployment, inadequate error handling (specifically the lack of null pointer checks), and a delayed rollback process. The incident highlights the importance of thorough testing, robust error handling, and effective rollback mechanisms in cloud infrastructure. While Google faced financial repercussions through SLA credits, the reputational damage could have a more lasting impact on its market share. The video also promotes PostHog and its AI-powered assistant, Max, as tools for building better products and preventing similar incidents.