PHX Networking Issue

We have continued to monitor the environment through the evening into this morning and have confirmed that no further issues are occurring. We will be marking this incident as resolved at this time. We expect to provide a postmortem and additional details in regards to this outage in the next two weeks.

At this time, all services appear to have been restored and we have received no further reports of ongoing issues. If you or your clients are still continuing to experience any issues, please submit a support ticket or reach out to us via phone to receive support on a case by case basis.

All services are in the process of being restored. Virtual machines should be coming up and a portion of them are already up. Networking issues have all be resolved, part of this issue was internal networking between management and production clusters and our storage network. The team is engaged on bringing up all servers in the proper order and confirming logins.

The datacenter team has resolved all outstanding issues on their end. We have also resolved an additional issue and are in the process of bringing the affected virtual machines back online. As virtual machines come back online and are confirmed to be operational, we will be issuing another status update.

We are actively investigating this issue. Currently, we’re experiencing difficulties re-establishing communication between the hosts and storage devices. Our team is working closely with the datacenter to resolve this as swiftly as possible.

We are seeing networking being restored at this time and are waiting for everything to finish coming back to begin verifying functionality.

We have identified the issue and are working with the datacenter to restore networking services.

We have received reports of VM's becoming unresponsive and are working to identify and resolve this issue with datacenter engineers as quickly as possible. We will be posting updates as we get them.