Outage in Fly.io

Delayed state updates: expect more proxy retries for instances of new deployments

Resolved Major
March 04, 2023 - Started about 2 years ago - Lasted 1 day
Official incident page

Need to monitor Fly.io outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Fly.io, and never miss an outage again.
Start Free Trial

Outage Details

An ongoing upgrade is causing delayed app instances state propagation
Components affected
Fly.io Routing Fly.io Deployments
Latest Updates ( sorted recent to last )
RESOLVED about 2 years ago - at 03/06/2023 03:11AM

This incident has been resolved.

MONITORING about 2 years ago - at 03/05/2023 10:34PM

State is a lot more consistent now and most problems should be resolved. Working on final consistency fixes.

MONITORING about 2 years ago - at 03/05/2023 04:12PM

We are now restoring missing state information from every server, 1 by 1.

MONITORING about 2 years ago - at 03/05/2023 01:51PM

We are aware some apps are still experiencing issues and are continuing work to fix state inconsistencies causing proxy timeouts.

MONITORING about 2 years ago - at 03/05/2023 04:45AM

There are still some issues with older deployments, we are still cleaning up the state.

MONITORING about 2 years ago - at 03/05/2023 02:26AM

A fix has been implemented and we are monitoring the results.

IDENTIFIED about 2 years ago - at 03/04/2023 08:46PM

We are continuing to work on a fix for this issue.

IDENTIFIED about 2 years ago - at 03/04/2023 07:01PM

Apps v1 deploys have been disabled as we're working on restoring a working state.

IDENTIFIED about 2 years ago - at 03/04/2023 05:24PM

We're rolling back this update

IDENTIFIED about 2 years ago - at 03/04/2023 02:46PM

An ongoing upgrade is causing delayed app instances state propagation

Latest Fly.io outages

New MPG clusters cannot be provisioned in FRA - about 16 hours ago
Errors (5xx, timeouts) in Fly.io dashboard - about 17 hours ago
IAD Managed Postgres control plane unavailability - 5 days ago
Some or all *.fly.dev subdomains are currently returning NXDOMAIN errors in IAD - 13 days ago
WireGuard connectivity into CDG is unavailable - 13 days ago

Stop Juggling Dozens of Status Pages – Monitor Them All in One Place

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4000 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook