Fly.io Outage History

Fly.io experienced elevated error rates on their List Machines API endpoint, specifically affecting applications with machines in the Singapore (sin) region. The API disruption lasted 38 minutes before the team successfully mitigated the problem and restored normal service.

Fly.io experienced a major API incident where users were unable to create new apps or update secrets on existing applications. The issue affected the platform's core functionality for 37 minutes before being resolved. The incident was successfully resolved without further details on the root cause.

Fly.io's log search functionality in Grafana was unavailable for 1.8 hours, with users encountering "failed to make http request: 502" errors when accessing logs from fly-metrics.net. App logs remained accessible through the `fly logs` command and the Fly.io dashboard during the outage. The incident was resolved after implementing a mitigation and monitoring the results.

Fly.io experienced a deployment issue where `fly deploy` commands were creating new machine instances instead of updating existing ones, causing applications to enter a mixed state. The problem affected the Deployments component and lasted 51 minutes before being resolved. Users could work around the issue by either removing the "processes = [ "app" ]" line from their fly.toml configuration or downgrading flyctl to version 0.4.40.

Fly.io experienced slow machine operations affecting a portion of machines in the IAD (Ashburn, Virginia) region due to network packet loss. The incident lasted 46 minutes, during which the company deployed a partial mitigation and monitored the Machines API for stability. The issue was resolved when network packet loss returned to normal levels.

Fly.io experienced a major incident where users encountered 500 errors when attempting to add new Github integrations or edit existing Github integrations through the dashboard. The issue specifically affected the "Launch an app from Github" feature and related settings, while existing integrations, flyctl deployments, and running apps continued to function normally. The incident was resolved after 1.4 hours once a fix was identified, implemented, and monitored.

Fly.io's web dashboard experienced 5xx errors and timeouts for 34 minutes, preventing users from accessing the dashboard interface. The engineering team identified the root cause and implemented a fix. The incident was fully resolved after a brief monitoring period to ensure stability.

Fly.io experienced increased latency in their Singapore (SIN) region for approximately 2.2 hours. The issue affected all services operating in the Singapore region. The incident was resolved after the team worked to address the latency problems.

Fly.io experienced a major incident with their Vault server that stores TLS certificates, causing failures in provisioning new certificates and connection issues for domains with uncached certificates. The SSL/TLS Certificate Provisioning component was affected for 31.6 hours. The issue was resolved after implementing a fix and monitoring the results.

Fly.io experienced networking issues in their Sydney (SYD) region that affected multiple central services for approximately 24 hours. The engineering team identified the root cause and applied a fix to restore normal operations. All services were confirmed to be working normally after the resolution was implemented.