Google Cloud June 2025: an unflagged feature, a policy with blank fields, and a global 3-hour authorization failure

Timeline

What happened

Times anchored to Google's published incident report, which gives the impact window in PDT. Most regions recovered by roughly 12:48 PDT; us-central1, the slowest region, took up to about 2 hours 40 minutes because Service Control lacked randomized exponential backoff. Intermediate steps are approximate where Google did not publish a minute-by-minute log.

Affected Services

Selected services and companies impacted

The list below is illustrative, not exhaustive. Service Control gates authorization for Google APIs, so its failure radiated to dozens of Google's own products and to third parties that depend on Google Cloud.

Root Cause

An unguarded feature and a policy with blank fields

Per Google's incident report, the seed was planted on 29 May 2025, when a new quota-policy check was added to Service Control. That change shipped with two gaps: it had no error handling for malformed input, and it was not protected by a feature flag. Google's own remediation notes that feature flags are used to enable features gradually, region by region, and that if this one had been flag-protected the fault would have been caught in staging. Instead the code sat inert but unguarded in production.

On 12 June, a quota policy change containing unintended blank fields was inserted. Service Control exercised its quota checks against that policy in each regional datastore, hit the missing fields on the new unguarded path, and dereferenced a null pointer. Rather than reject the bad input, the process crashed and entered a crash loop. Because the policy data was replicated globally within seconds, the same crash hit every region almost simultaneously, which is why authorization failed worldwide rather than in one region.

Detection was fast: engineers identified the cause within minutes and deployed a red-button to bypass the offending policy check, and most regions recovered in about two hours. Recovery lagged in us-central1, however, because Service Control did not have randomized exponential backoff, so as tasks restarted they overloaded the underlying infrastructure in a herd effect. Google throttled task creation and routed traffic to multi-regional databases to bring the region back safely, which is what stretched full resolution there to up to about 2 hours 40 minutes.

Economic Impact

No disclosed figure, so model the exposure instead

Google did not publish a cost for this outage, and, unlike the AWS October 2025 event that CyberCube priced at $38 million to $581 million in insured losses, no independent analyst issued a specific insured-loss estimate for the June 2025 Service Control incident. Rather than invent a headline number, the honest measure is the exposure: a global, roughly 3-hour authorization failure across dozens of Google Cloud and Workspace products during a weekday business window, plus the downstream hit to Cloudflare, Spotify, Discord, and OpenAI.

For the customers hit, the loss was not Google's SLA credit. GCP returns a percentage of the affected service's monthly fee, never a percentage of your revenue, so even a maximum credit is a fraction of the business loss from the same window. See our SLA credit asymmetry analysis for why the credit rarely covers the damage.

To translate the event into your own numbers, the downtime cost calculator applies your revenue and architecture to a 3-hour outage. A business running $100,000 per day of GCP-dependent revenue would face roughly $12,500 of exposure across a 3-hour total outage before any failover mitigation.

Architectural Lessons

A global control plane insulates no region

The June 2025 incident is a clean illustration that a global control plane is a concentration risk for everything behind it. Service Control exists so that authorization and quota decisions are consistent everywhere, but that also means one bad policy replicated globally can fail authorization in every region at once. Running in multiple GCP regions gave no protection here, because all of them shared the same control plane and the same bad data.

Three lessons recur. First, new code in a control plane must fail safe: a missing-field policy should have been rejected, not allowed to crash-loop the process. Second, feature flags exist to catch exactly this class of fault in staging and to limit blast radius by enabling a change region by region; skipping them turned a staging-catchable bug into a global outage. Third, recovery mechanics matter as much as the fix: without randomized exponential backoff, the restart itself became a second incident in us-central1. Google's remediation committed to all three.

For how this event compares with the AWS us-east-1 and Azure Front Door failures of 2025, see the AWS vs Azure vs GCP outage comparison and the full GCP outage history. The downstream Cloudflare failure is covered in the Cloudflare outage history. For the cost-benefit math on multi-region and failover, see our business case builder.

Frequently Asked

Common Questions