Las Vegas Network

Future updates will be covered on the "Apollo - Las Vegas Datacenter Connectivity" incident page.

Last night, we identified a core network problem at our Las Vegas data center. Initially, we put Vegas into 503 mode as a temporary measure while we investigated the issue. This morning, however, we discovered that some devices—specifically those with phone configurations or network setups that did not include proper SRV records—continued trying to connect to a server in 503 mode.

To address this, we removed the 503 status since the core network problem was affecting only a small subset of operations. At the same time, we ran a script to automatically shift all devices that are currently preferred to Las Vegas over to our NJ or Florida data centers. Please note that if your phone is hardcoded to Las Vegas, it will not be redirected to the new sites. Also while moving the phones to their new data center the phones will experience a reboot, this reboot will not be intrusive if you are on an active call.

We are actively conducting emergency maintenance on the Las Vegas data center. The impact of this should be minimal, as devices are being moved to our NJ and Florida facilities. Once the migration is confirmed, we will temporarily remove the Las Vegas data center from our Netsapiens manifest. This will allow us to focus on fully resolving the core network issue. Once the fix is validated, we will reintegrate Vegas back into production.

We believe this approach will not only address the current incident but also help mitigate related queuing issues with SBus between Vegas and NJ that could cause intermittent connectivity problems.

A full root cause analysis will be published once everything has been resolved. Thank you for your patience and support as we work through these issues.