Use Cases
Software Products MSPs Schools Development & Marketing DevOps Agencies Help Desk
 
Internet Status Blog Pricing Log In Try IsDown for free now

Outage in Quad9

North America + Australia - Major Outage due to BGP Hijack by CenturyLink (Lumen)

Resolved Minor
August 22, 2024 - Started over 1 year ago - Lasted less than a minute
Official incident page

Incident Report

CenturyLink (AS209), a network belonging to Lumen (AS3356) hijacked and advertised hundreds of IPv4 routes across many networks (ASNs), causing them to intercept and blackhole Quad9 and other networks' traffic in various locations globally. This seems to have primarily affected networks in North America and Australia, as they believed this was a "new" and "best" path to reach Quad9. A few, smaller networks in Europe may have also been affected if Lumen was their primary upstream and they do not peer at any internet exchange points. This was detected by Qrator, and likely other BGP Monitoring services, and there will likely be significantly more information/news to follow publicly in the coming hours and days: https://x.com/Qrator_Radar/status/1826600702996873489 In this case, there was nothing Quad9 could do, as the internet is still vulnerable to these types of misconfiguration issues. Quad9 is RPKI validated, so the affected networks need to implement RPKI filtering to protect their customers. Note that this only affected our IPv4 address. If your network supports IPv6, we recommend making sure configuring our IPv6 addresses, as this would've avoided a total DNS outage for your networks/devices. This may have affected other networks in other regions, but the largest, affected networks were: * Comcast (US) * Century Link (US) * Telstra Domestic (AU/) * Bell (CA) * Rogers (CA) Issue Start: 12:15 UTC, Aug 22nd. Issue Stop: 12:45 UTC, Aug 22nd.

Need to monitor Quad9 outages?

  • Monitor all your external dependencies in one place
  • Get instant alerts when outages are detected
  • Be the first to know if service is down
  • Show real-time status on private or public status page
  • Keep your team informed
Latest Updates ( sorted recent to last )
RESOLVED over 1 year ago - at 08/22/2024 03:16PM

CenturyLink (AS209), a network belonging to Lumen (AS3356) hijacked and advertised hundreds of IPv4 routes across many networks (ASNs), causing them to intercept and blackhole Quad9 and other networks' traffic in various locations globally.

This seems to have primarily affected networks in North America and Australia, as they believed this was a "new" and "best" path to reach Quad9. A few, smaller networks in Europe may have also been affected if Lumen was their primary upstream and they do not peer at any internet exchange points.

This was detected by Qrator, and likely other BGP Monitoring services, and there will likely be significantly more information/news to follow publicly in the coming hours and days:
https://x.com/Qrator_Radar/status/1826600702996873489

In this case, there was nothing Quad9 could do, as the internet is still vulnerable to these types of misconfiguration issues. Quad9 is RPKI validated, so the affected networks need to implement RPKI filtering to protect their customers.

Note that this only affected our IPv4 address. If your network supports IPv6, we recommend making sure configuring our IPv6 addresses, as this would've avoided a total DNS outage for your networks/devices.

This may have affected other networks in other regions, but the largest, affected networks were:

* Comcast (US)
* Century Link (US)
* Telstra Domestic (AU/)
* Bell (CA)
* Rogers (CA)

Issue Start: 12:15 UTC, Aug 22nd.
Issue Stop: 12:45 UTC, Aug 22nd.

The Status Page Aggregator with Early Outage Detection

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 6020 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook