Cloudflare suffered a major global outage earlier this week after an internal configuration change caused its network to intermittently fail for several hours. Users worldwide encountered Cloudflare 5xx errors when trying to access websites, applications, and services behind the company’s infrastructure.
Many websites including several major online services such as X (formerly Twitter), ChatGPT and Spotify also experienced disruptions.
Cloudflare confirmed that the disruption was not the result of a cyberattack. It was triggered by a permissions update to a ClickHouse database cluster. The change caused the system that generates Bot Management’s configuration “feature file” to create duplicate entries, doubling the file’s size.
That oversized file propagated across Cloudflare’’’s global edge, where proxy software attempting to load it exceeded a built-in limit and crashed. Because the file refreshed every few minutes, different versions alternated between valid and invalid, causing the entire network to repeatedly fail and recover before eventually remaining in a degraded state.
A large DDoS attack was initially suspected due to the erratic symptoms, but engineers later isolated the issue and halted distribution of the malformed file. A known-good version was redeployed restoring core traffic flow before full service recovery was completed.
Multiple services were affected during the outage, including Cloudflare’s CDN, security layers that rely on bot scoring, Workers KV, Turnstile authentication, and Cloudflare Access. Some users were temporarily unable to log in to the dashboard, and bot-dependent rules briefly produced inaccurate results.
Cloudflare says it was its most severe outage since 2019 and is implementing additional safeguards, including stricter validation for internal configuration files and broader fail-safe mechanisms to prevent similar events.
Leave a Reply