Outage in Fly.io

Consul cluster outage

Resolved Major
March 15, 2023 - Started about 2 years ago - Lasted about 8 hours
Official incident page

Need to monitor Fly.io outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Fly.io, and never miss an outage again.
Start Free Trial

Outage Details

Our Consul cluster is experiencing an outage. This impacts queries to our API, including creating and modifying apps, as well as incoming network requests for recently deployed apps.
Components affected
Fly.io Routing Fly.io API Fly.io Deployments
Latest Updates ( sorted recent to last )
RESOLVED about 2 years ago - at 03/16/2023 03:28AM

This incident has been resolved.

MONITORING about 2 years ago - at 03/16/2023 12:58AM

Our Consul cluster is now stable again. We've re-enabled Nomad deployments and are monitoring for any further issues.

IDENTIFIED about 2 years ago - at 03/16/2023 12:04AM

The Consul cluster rebuild is nearly completed and we are working to restore service.

IDENTIFIED about 2 years ago - at 03/15/2023 11:53PM

The Consul cluster rebuild is nearly completed and we are working to restore service.

IDENTIFIED about 2 years ago - at 03/15/2023 10:42PM

We are working to build a new Consul cluster with 10x the RAM. We aren't yet sure, but believe a routine DNS change might have created a thundering herd problem causing Consul servers to immediately increase RAM usage by 500%. This is not ideal.

IDENTIFIED about 2 years ago - at 03/15/2023 09:10PM

We're still working on recovering our Consul cluster.

MONITORING about 2 years ago - at 03/15/2023 07:20PM

Our Consul cluster is experiencing an outage. This impacts queries to our API, including creating and modifying apps, as well as incoming network requests for recently deployed apps.

Latest Fly.io outages

IAD Managed Postgres control plane unavailability - 2 days ago
Some or all *.fly.dev subdomains are currently returning NXDOMAIN errors in IAD - 10 days ago
WireGuard connectivity into CDG is unavailable - 10 days ago
Upstream network outage in MAD - 12 days ago
Fly.io dashboard down - 16 days ago

Stop Juggling Dozens of Status Pages – Monitor Them All in One Place

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4000 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook