Railway Outage History

Railway experienced elevated error rates for SSL certificate issuance affecting custom domains, where users encountered failures when adding or renewing certificates. The issue was caused by an upstream incident with Railway's certificate provider. The incident lasted 1.2 hours and was classified as minor severity.

Railway experienced a 1-hour incident where builds were queuing and taking longer than normal to start due to a bug in a recent builder image. The issue was resolved by rolling back to an older builder image, and build queues began shrinking as the system recovered.

A single unhealthy proxy in the EU West region caused intermittent connection issues, resulting in users experiencing connection resets and failed requests. The affected proxy was identified and removed from rotation, with traffic redirected to healthy proxies. The incident lasted 1.3 hours and connections returned to normal once the problematic proxy was taken offline.

Railway experienced slow build deployments for 34 minutes, primarily affecting the US West region due to an upstream provider issue. The service degradation caused builds to take longer than usual to complete. The issue was resolved after Railway worked with their upstream provider to address the root cause.

Railway experienced edge network connectivity issues in their Singapore/Southeast Asia region for 31 minutes, causing users to encounter failed requests and DNS resolution errors when accessing services deployed in that region. The edge routing problems have been resolved and the service is now monitoring to ensure stability.

The Railway service experienced elevated latency in the US East edge network for 31 minutes, causing requests to take longer than normal to complete. The issue was traced to a problem with an upstream CDN provider, which was resolved by removing the affected provider from the edge network. Performance returned to normal after implementing this mitigation.

Railway experienced degraded performance affecting stateful services with attached volumes, causing elevated latency and slower disk I/O for approximately 1.3 hours. Stateless services and new deployments without volumes remained unaffected during the incident. A fix was implemented and performance returned to normal levels.

Railway's build pipeline experienced significant delays and slowdowns for 5.5 hours due to an upstream issue with GitHub services, causing builds to queue for extended periods and take longer than usual to complete. The incident affected build processing times and created a backlog as elevated build volumes overwhelmed the system. Engineers scaled the build pipeline and monitored recovery as GitHub resolved their upstream issues, with build times gradually returning to baseline levels.

Railway experienced outbound networking timeouts to Azure services due to a problematic change involving static IP egress gateways. The incident lasted 1.4 hours and was resolved by rolling back the problematic change, with recovery confirmed during monitoring.

Railway experienced a minor service incident due to a Dockerhub Registry outage that caused Docker image pulls from Dockerhub to fail. The outage lasted approximately 1.2 hours and was resolved. The incident was classified as minor impact to Railway's services.