Railway Outage History

Railway experienced slow build and deployment times in the US-West region due to elevated image pull times, causing new deployments to take longer than usual or appear stuck in progress. The issue was isolated to part of the US-West region while other regions and existing running services remained unaffected. The problem was resolved after 46 minutes with image pull times returning to normal levels and builds/deployments progressing as expected.

Railway experienced a build log delivery issue lasting 4.5 hours where build logs were missing or incomplete for deployments. The underlying cause was identified and resolved with a deployed fix, though some build logs from the affected timeframe may be permanently missing. Running applications were not impacted throughout the incident.

Railway's GitHub integration experienced intermittent deploy failures for 3.7 hours due to upstream authentication issues with GitHub's app installation tokens. Users encountered "Bad credentials" and "repository not authorized" errors when attempting to deploy from GitHub repositories. The issue was resolved as GitHub mitigated their authentication system problems, allowing deploys to recover automatically.

Railway experienced intermittent connectivity issues affecting both railway.com and user-deployed services, with users encountering SSL handshake failures, 400 errors, and brief service unavailability. The engineering team identified the root cause and deployed a fix while isolating the affected workload to restore connectivity. The incident was resolved within 27 minutes with ongoing monitoring to ensure system stability.

Railway experienced a 10.4-hour incident where builds and deployments were significantly delayed due to a backlog in the build queue and an issue with builder storage. During the incident, builds and deployments for hobby plans were temporarily paused multiple times to reduce system load, while Pro plan services continued processing with delays. The incident was resolved by scaling up build workers, implementing tuning adjustments, and deploying a fix for the builder storage issue, allowing the queue to fully drain and normal processing to resume.

Railway experienced a 5.6-hour service disruption caused by Google Cloud blocking their account, which made Railway services unavailable and affected the dashboard, API, and internal network control plane. Users experienced errors including "no healthy upstream," "unconditional drop overload," login failures, and inability to access the dashboard. Railway restored access by working directly with Google Cloud support and gradually brought workloads back online, with some services requiring manual redeployment from the dashboard or CLI to fully recover.

Railway experienced a major service disruption lasting 7.3 hours that affected core platform functionality. Users encountered multiple critical errors including "no healthy upstream" and "unconditional drop overload" messages, along with login failures and complete inability to access the dashboard. The widespread outage impacted Railway's entire service infrastructure before being resolved.

Railway experienced a 57-minute outage where all builds and deployments failed to process due to a networking issue with an upstream provider. The incident affected the entire build and deployment pipeline, preventing any new deployments from going out and causing queued builds to stop progressing. All deployments were re-enabled after the networking issue was resolved, and the queued builds and deployments resumed processing normally.

Railway experienced elevated latency and failed requests affecting services with static IPs enabled across all regions for 51 minutes. Users encountered slow response times and connection failures on outbound connections from these services. The issue was identified and resolved with a fix, followed by monitoring to confirm full recovery.

Railway experienced a 49-minute incident where metrics and logs were delayed or failed to load for some users. The issue was identified and resolved, with metric and log request times returning to normal latency. The service team monitored the recovery to ensure full restoration.