Kalix Downtime Due To Azure WestUS Outage

There has been no ongoing issues with Kalix since the component recovered.

The affected component has just been restored, so Kalix is operational again.

We are monitoring to make sure that all systems are functioning as expected.

The service that is causing the current Kalix outage is experiencing downtime in the West US region of Azure due to a sudden power outage, and this issue is still being worked on. Further updates can be seen here:

https://azure.status.microsoft/en-us/status

While the power outage has been solved, it seems like the recovery is still underway.

Attempts to bypass the component were not successful, as spinning up alternatives is also affected by the downtime in the West US region. Bypassing the region entirely will take more time as our databases would have to be replicated and recreated, so we are waiting for the West US region to recover.

We were able to find the underlying cause of the downtime - most of our infrastructure/databases are live, but we have one service that manages our backend queues that is has been affected by the downtime.

We are working with our provider to bring this component back up, or will possibly create an alternative component to use instead.

After more investigation we have found that this is not a database issue, but rather that the application is not properly starting up (but after the database connection step). We are currently pushing different versions to try and work out the spot that is blocking the start-up.

We have managed to narrow down the issue to the connection between the API server and the database, but we are having trouble working out why these two components are not able to talk to each other. Our current theory is that the power outage affected the network. We are continuing to investigate.

We have found out that the underlying issue was caused by a power failure in the datacenter. It is running on backup but there were some systems that did not recover correctly.

Most of our components are live but we are still having trouble with our API which manages the backend of Kalix, this might be affected by the reduced capacity of the datacenter but we are looking to see if there is a secondary issue.

There are issues logging in to Kalix because the authentication servers are not running.

There is currently an outage at our main provider location (Azure West US), causing downtime in Kalix. We are attempting workarounds by starting new servers, but the issue appears to be communication within the datacenter. We will continue to monitor the status of the servers and take further action if it looks like this issue is going to be extended.