Railway Outage History

A hardware failure on a single host in Railway's EU West region caused service unavailability and deployment failures for a subset of users over 1.7 hours. The incident affected compute infrastructure, with services using attached storage and databases experiencing longer recovery times. Railway restored the affected infrastructure and services gradually recovered, with users advised to redeploy if their services didn't automatically recover.

Railway experienced outbound networking issues affecting a subset of hosts, with egress traffic failures primarily concentrated in the Asia region starting at 0620 UTC. The incident lasted 4 minutes and was classified as minor severity. The service was investigating the issue at the time of the last update.

Railway experienced networking failures affecting a subset of deployments, preventing them from reaching the internet. All networking functionality was impacted for the affected deployments. The issue was resolved after 36 minutes when Railway identified the root cause, implemented a fix, and began monitoring the results.

Railway experienced a logs loading issue that caused elevated delays for users trying to access their application logs. The incident lasted 2.1 hours with continued degradation even after an initial fix was implemented. The team worked on multiple fixes throughout the incident to resolve the slow loading times.

Railway experienced an issue where some code repositories failed to snapshot properly when using metal builders. Non-metal builders continued to function normally during the incident. The issue was resolved with a fix implementation after 28 minutes of investigation and monitoring.

Railway experienced a minor incident where Dockerfile build configurations failed to resolve properly, causing builds to fail. The issue was identified within 12 minutes and lasted a total of 24 minutes. The team identified the root cause and implemented a fix to restore normal build functionality.

Railway experienced deployment slowness for 5.6 hours, with deployments taking longer than normal and experiencing extended queue and build times. The incident primarily affected trial deployments initially, but elevated system load eventually required temporarily pausing Free, Trial, and Hobby deployments while Pro deployments were restored first. The issue was resolved by eliminating the deployment pipeline slowness and gradually re-enabling deployments for all plan types as system backpressure recovered.

Railway experienced a deployment issue where new builds appeared stuck, failed to complete, or progressed slowly for 1.3 hours. The incident affected new deployments only, while existing running services continued to operate normally. The team temporarily disabled hobby, free, and trial deploys, identified the root cause, and deployed a fix to resolve the issue.

Railway experienced TCP proxy connectivity issues primarily affecting the US East and Asia regions for 20 minutes. The incident impacted users' ability to establish TCP proxy connections in these regions. Railway implemented a fix and restored service after identifying and resolving the underlying connectivity problems.

Railway experienced a deployment issue that caused a large portion of deployments to fail and created slowdowns across their deployment and build systems. The incident affected all users, with hobby, free, and trial plan deployments temporarily paused during the resolution process. The team identified the root cause, deployed a fix, and restored normal service including re-enabling all deployment tiers after 1.1 hours.