AWS us-east-1 October 2025: a DynamoDB DNS race condition and a 15-hour cascade

Timeline

What happened, hour by hour

Times anchored to AWS's official post-incident summary (which reports PDT) and cross-checked with the ThousandEyes outage analysis. ET conversions added for readability.

Affected Services

Selected major customers impacted

The list below is illustrative, not exhaustive. Downdetector logged problems across more than 1,000 services and millions of user reports globally. The breadth reflected DynamoDB's position as a hidden dependency beneath thousands of applications hosted in or routed through us-east-1.

Root Cause

An empty DNS record the automation could not repair

Per AWS's official summary ("Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region"), DynamoDB's DNS records are maintained by an automated system with two roles: a DNS Planner that generates new plans and DNS Enactors that apply them. A latent race condition let two Enactors run concurrently. One Enactor ran slowly while a second applied a newer plan; a stale-plan check then allowed the slow Enactor's older plan to overwrite the newer one, and cleanup automation deleted the plan it considered obsolete. The result was an empty DNS record for the regional endpoint dynamodb.us-east-1.amazonaws.com, which the automation could not self-repair.

With the endpoint unresolvable, every service that depended on DynamoDB in us-east-1 began returning errors. DynamoDB itself was impaired for roughly three hours, from about 11:48 PM PDT on 19 October to 2:40 AM PDT on 20 October, until engineers manually restored the record and disabled the DNS Planner and Enactor automation worldwide while they fixed the race condition.

The deeper damage came from the cascade. EC2's DropletWorkflow Manager (DWFM) uses DynamoDB to track the leases that keep physical capacity assigned to instances. While DynamoDB was unreachable, those leases expired fleet-wide. When DynamoDB recovered, DWFM had to re-establish every lease at once and fell into congestive collapse: it could not finish the re-leasing work before it timed out and had to restart. New EC2 launches returned "insufficient capacity" errors, and Network Load Balancer health checks failed for instances that launched without network state. The EC2 and networking tail therefore lasted hours after DynamoDB itself was healthy, pushing broad recovery to about 1:50 PM PDT on 20 October.

Economic Impact

$38M to $581M in insured losses, and far more uninsured

AWS does not disclose customer-impact figures. The most-cited independent estimate comes from CyberCube, whose Cyber Aggregation Event Response Service put preliminary insured losses in a range of $38 million to $581 million for the event it nicknamed "Amazonk". CyberCube estimated that about 70,000 organizations were affected, including over 2,000 large organizations, and assessed the event as only a moderate insurance impact (a low- to mid-single-digit loss-ratio effect for cyber insurers).

Insured losses are a floor, not the total. They exclude uninsured revenue loss, lost productivity, SLA-credit payouts, and the engineering cost of recovery, none of which AWS or its customers disclose in aggregate. For context on why the SLA credits paid out were small relative to those losses, see our SLA credit asymmetry analysis: AWS returns 10 to 30 percent of the affected service's monthly fee, never a percentage of your revenue.

To translate the event into your own numbers, the downtime cost calculator applies your revenue and architecture to a 15-hour outage. A business running $100,000 per day of AWS-dependent revenue would face roughly $62,500 of exposure across a 15-hour total outage before any failover mitigation.

Architectural Lessons

When a single service is a dependency for everything else

The October 2025 incident is the clearest recent illustration that hyperscaler outages are rarely a single service failing in isolation. DynamoDB is a dependency beneath EC2 control-plane operations, Lambda, IAM and STS token issuance, Amazon Connect, and dozens of managed services in us-east-1. Some services also expose global endpoints whose control plane lives in us-east-1, so a regional failure radiates outward even for customers who believe they are elsewhere.

Three practical lessons recur. First, recovery from a cascading failure is harder than the failure: the DWFM congestive-collapse tail meant EC2 stayed degraded for hours after the original DNS fault was fixed, because re-establishing state at full scale overwhelmed the system. Second, "multi-region" on a diagram is not the same as regional independence in operation; only game-day failover testing reveals the hidden us-east-1 dependencies in IAM, STS, and deployment tooling. Third, DNS and other control-plane automation deserve the same staged-rollout and safety checks as application code, because a single automated action wiped a record that no customer could route around.

For the cost-benefit math on multi-region active-active versus single-region with strong backup, see our business case builder, and for how this event compares with the December 2021 and February 2017 us-east-1 incidents, see the full AWS outage history.

Frequently Asked

Common Questions