Dashboard loads slowly and users cannot start workspaces

We are pleased to confirm that the issue has been successfully resolved, and all system functionalities have been restored to normal operation.

We've restored the system, workspace starts and the dashboard experience are normal again.

We had to rollback a change that was putting excessive load on the system, it was causing API request duration to be too high.

We've applied mitigation, this allows for workspace starts. However, the API request and error rate is still elevated, which is causing the dashboard and workspace start experience to be slow. We're still investigating.

We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.