Elevated Error Rates: EMEA (Frankfurt)

We have been able to confirm resolution on the issue. Teams will be taking additional actions today and continue to monitor. For any customers impacted a comprehensive RCA will be attached to cases.

Due to the long duration and nature of the corrupt data, we are going to monitor for an extended time to ensure all clear.

Health checks are passing. Monitoring.

Seeing major improvement after last mitigation step. Monitoring health checks.

We are continuing to work on a fix for this issue.

The root cause of the issue is due to corrupt data in a cache. Attempts to bypass the cache and/or flush have not been successful. We apologize for the duration and are working diligently to resolve.

We are continuing to work on a fix for this issue.

Mitigation steps are still in progress.

Maintenance modes are still being observed but at lesser rates than previously reported. Presence status continues to experience elevated error rates. Additional mitigation steps are in progress.

Debugging has uncovered a condition previously not observed. Next steps are being identified. We understand the inconvenience this is causing for users and are working diligently to resolve it.

No change in status. Debugging continues.

Teams have cycled instances and are trying to bypass the cache to see if that provides relief. Issue appears to be corrupt data after a rollback was performed earlier.

Teams completed cycling instances and while additional improvement observed, errors persist. Next update top of the hour.

Teams are continuing investigation to address error rates. Recent mitigation had some affect, but errors have not improved at a satisfactory rate. Current impact is latency with presence status for some users. Next update bottom of the hour.

Mitigation steps have been applied. Some errors still remain and users may continue to experience latency with presence status changes. Next steps are being determined. Next update top of the hour.

Issues with login (maintenance mode) have largely subsided. There have been reports of latency with presence status. Teams are determining next steps. Next update bottom of the hour.

Elevated error rates have been observed in the EMEA (Frankfurt) region. Some users have reported issues with new logins. After retry, many of those users are able to successfully login. Teams have been engaged for triage and mitigation. Next update bottom of the hour.