Outage in Fly.io

Consul/Nomad outage interfering with app instance scheduling

Resolved Minor
October 27, 2022 - Started almost 3 years ago - Lasted 1 day
Official incident page

Incident Report

We are currently investigating an issue with consul and nomad, which is causing issues with deployment monitoring. Deployments are still happening.

One place to monitor all your cloud vendors. Get instant alerts when an outage is detected.

Try IsDown risk-free 14-day free trial · No credit card required
Latest Updates ( sorted recent to last )
MONITORING almost 3 years ago - at 10/28/2022 02:30PM

We have cleaned up most lingering issues from the Consul/Nomad outage and are monitoring to ensure that peoples' apps recover properly.

IDENTIFIED almost 3 years ago - at 10/28/2022 01:42PM

We are still working on this issue. Deploys, restarts, and rescheduling of apps are all currently affected by this.

IDENTIFIED almost 3 years ago - at 10/28/2022 11:00AM

We are having continued issues with our consul and nomad clusters. We are working on restoring them.

MONITORING almost 3 years ago - at 10/28/2022 07:29AM

We've repaired our Nomad and Consul clusters and are working through a backlog of queued operations. Deployments and restarts will continue to be delayed ~5-30m until the queue is empty.

INVESTIGATING almost 3 years ago - at 10/28/2022 03:01AM

Deployment monitoring is erroring in most cases. Deployments themselves are working through a queue. It's currently taking upwards of 30 minutes for deploys to take effect.

INVESTIGATING almost 3 years ago - at 10/28/2022 02:16AM

We are currently investigating an issue with consul and nomad, which is causing issues with deployment monitoring. Deployments are still happening.

Latest Fly.io outages

IPv6 networking issues on some hosts - 2 days ago
"unauthorized" errors when starting some Fly builds - 5 days ago
Intermittent 500 errors on the dashboard - 7 days ago
Registry pulls using a local docker client return an error - 11 days ago
Elevated Packet Loss - 12 days ago

The Status Page Aggregator Built for IT Teams

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4522 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook