AskCody Platform - Operational (Monitoring)

Since the 25th of January, when Microsoft implemented their fix, page load times have continued to be stable, and no errors related to the incident first reported on the 19/01/23 have been registered.

Following this monitoring period, we are proceeding to mark this incident as resolved.
A full postmortem will follow as soon as we can deliver it, explaining in as much detail as possible what has transpired.

Page load times continue to be stable, and since the previous update, no errors related to the incident first reported on the 19/01/23 have been registered.
For good measure, we will monitor the status of the platform throughout the weekend.

On Monday our intention is to set this incident to a resolved status. A full postmortem will follow as soon as we can deliver it, explaining in as much detail as possible what has transpired.

Page load times continue to be stable, and since the previous update, no errors related to the incident first reported on the 19/01/23 have been registered.
The other open incident, posted at the time of the recent Microsoft incident will be marked as resolved, allowing for any relevant updates to follow in this thread. https://status.askcody.com/incidents/7d7j5mw14cfp

A full post-mortem will follow as soon as we can deliver it, explaining in as much detail as possible what has transpired.

Unless any relevant information related to this incident presents itself, the next update will follow tomorrow at 11:00 AM.

Following the fix Microsoft implemented yesterday, we have continued to experience platform stability.
Page load times are stable, and since the previous update, no errors related to the incident first reported on the 19/01/23 have been registered.

A full post-mortem will follow as soon as we can deliver it, explaining in as much detail as possible what has transpired.

Unless any relevant information related to this incident presents itself, the next update will follow at 15:00.

We are continuing to monitor the performance of the AskCody platform.
Page load times are stable, and since the previous update, no errors related to the incident first reported on the 19/01/23 have been registered.

We will continue to update as soon as new information is available, but no later than 07:30 (UTC+1) on the 26/01/23.

We are continuing to monitor the performance of the AskCody platform.
Page load times are stable, and since the previous update, no errors related to the incident first reported on the 19/01/23 have been identified.

We will continue to update as soon as new information is available, but no later than 21:00 (UTC+1).

Dear Customer,

We are seeing steady load times, since Microsoft started acting on the issue and started a roll-back, according to the issues announced by Microsoft this morning: https://azure.status.microsoft/en-gb/status & https://status.office365.com/.
Our entire platform has been scanned, measured, monitored, analyzed and services rebooted, as told in our morning status, and the above provides us with the confidence in that we are certain this is the cause.

We will keep this in a monitoring status and keep it here until we feel certain the implemented measures from Microsoft are in fact in place and that we are back on track with the incident first reported on Thursday the 19th of January 2023.

We are putting everything back to operational statuses because all performance metrics indicate stable performance. However, in light of the fact that this incident has been active for 6 full days, we will be extra diligent and cautious, and we will make sure to update you once we feel more confident on whether or not we are operational.

A full post-mortem will follow as soon as we can deliver it, explaining in as much detail as possible what has transpired.

Best,
Customer Experience Director, Kim Lunden Jensen

We will continue to update as soon as new information is available, but no later than 17:00 (UTC+1).

The platform is generally accessible. Occasional slow load times and timeout errors may however still be experienced.
We are continuing to investigate the issue related to the performance of the AskCody platform, as well as the impact of the Microsoft Incident on AskCody. https://status.office365.com/

We will continue to update as soon as new information is available, but no later than 14:00 (UTC+1).

We are currently focusing on: AskCody Platform Degraded Performance - related to Microsoft M365 dependencies.
https://status.askcody.com/incidents/7d7j5mw14cfp

We will continue to update as soon as new information is available, but no later than 12:00 (UTC+1).

We are continuing to investigate this issue.

Dearest Customer,

We are very aware that every passing minute, hour, and day that the AskCody Platform is not available to the extent needed to perform accordingly with your expectations, is painful and troublesome, for you, for your operations and your users. We are deeply sorry for the impact it is causing your business.

Respectfully, we can now share some more context and information on the issues and challenges that have impacted our Platform and its availability since Thursday last week.

A full Post-Mortem will be done and shared as soon as possible afterwards. Right now, we are prioritizing a solution and a fix first; then we settle and evaluate afterwards.

This information is simply to clarify what we know, by now.

What we know so far:

On Thursday the 19th, Microsoft shut off Basic Authentication for Exchange Online in Europe - Basic Authentication Deprecation in Exchange Online – Time’s Up - Microsoft Community Hub (https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-deprecation-in-exchange-online-time-s-up/ba-p/3695312), Even though we, AskCody and our Customers, prepared for this, moved everyone to Modern Authentication, that event initiated a series of other events leading to the outage and extreme slow response times to the AskCody Platform to Customers in Europe. No North American customer running on our US Data Center has been impacted, since Basic Auth hasn’t been shut down yet by Microsoft in such region.

Since this change to Microsoft Exchange in Europe, on Thursday the 19th, every time our Platform connects with Microsoft Exchange (even though we’re using Modern Auth, SSO or accessing data via Microsoft Graph that theoretically shouldn’t be impacted), the connection times-out or is put on hold in a que until getting a 401 response. Our build in algorithms forces our Platform to retry to connect automatically, which again fails as a consequence of the secondary, indirect, or cumulative effects of Microsoft shutting down Basic Auth.

Despite the shutdown of Basic Auth, it should not have impacted any customer using Modern Auth, but for some reason, it has caused connections with Microsoft Exchange to timeout, and simply lead to way to high load times on our Platform, again leading to the Platform being unavailable due to the high load, generated by false, unauthorized requests to Exchange.

How this can happen, and the full extent of the consequences, are still being investigated and will be shared the second we know more. The issue has been escalated and raised with Microsoft, and right now we have an open case running with Microsoft to get knowledge on what has changed.

Our entire platform has been scanned, measured, monitored, analyzed and services rebooted. Since Thursday, our Application Layer, our infrastructure, and our storage have been fully functioning and running – despite the Platform being inaccessible from an end-user perspective. The reason for the inaccessibility is due to the high load times, forced by bad request to Exchange, forced by indirect, secondary effects on Microsoft shutting down Basic Auth. Therefore, our normal recovery procedures for rolling back to the last know steady state, has not been a solution, since our entire application layer and infrastructure has been running as in normal operations.

By Friday and Monday, we rolled out, and updated, how we access data in Microsoft Exchange to leave out and remove all bad requests to ensure no connection, good or bad, just lies in a queue, forcing overload to the Platform. This has already had a significant impact on availability and speed.

Today, we further scaled the updates to mitigate some potential bottlenecks, that we discovered, with the maximum pressure that has been put on that Platform due to false request by Exchange in Europe.

In combination with these series of events and the consequences of these, our Azure Load Balancer has been off too, causing another series of issues in a perfect storm, where traffic has been routed to the wrong datacenter. This has been resolved too, and we are awaiting a response from Microsoft on how this could happen, as well.

Monitoring everything in the afternoon on Tuesday indicates that we have mitigated and resolved the challenges, to get us back to normal operations. We will not conclude anything until we’ve seen normal operations for a full workday.

This is not an attempt to push blame or instigate anything towards Microsoft. This is for you, our customers, to try and help you with what we know so far, and although nothing is set in stone it is what we know so far.

A full post-mortem will be done when everything is fully solved.

We will continue to update you with the same frequency, until it is all resolved, and we will make sure to provide you with a clear explanation once resolved.

In the meantime, please do not hesitate to reach out, if you feel we can add anything that helps you communicate this and helps you relieve some of the pressure we know this causes. We are happy to jump on to any call 24/7 to explain the situation and what we are doing to solve it, based on our knowledge and insight available.

Everyone is working tirelessly to resolve the situation and we hope to deliver a full, available solution soon. Right now, we are monitoring the latest fixes deployed on Monday and Tuesday, to understand the impact. We can see repones time is going dramatically down, and the number of bad requests too,

The second we have any news; we’ll share it openly with you.

Best,

Kim Lunden Jensen, Director of Experience.




As for status on the platform:
The platform is generally accessible. Occasional slow load times and timeout errors may however still be experienced.
We are continuing to investigate the issue related to the performance of the AskCody platform.

We will continue to update as soon as new information is available, but no later than 09:30 (UTC+1).

The platform is generally accessible. Occasional slow load times and timeout errors may however still be experienced.
We are continuing to investigate the issue related to the performance of the AskCody platform.

We will continue to update as soon as new information is available, but no later than 07:10 (UTC+1) on the 25/01/2023.

The platform is generally accessible. Occasional slow load times and load errors are possible.
We are continuing to investigate the issue related to the performance of the AskCody platform.

We will continue to update as soon as new information is available, but no later than 21:00 (UTC+1).

The platform is generally accessible. Occasional slow load times and timeout errors may be experienced.
We are continuing to investigate the issue related to the performance of the AskCody platform.

We will continue to update as soon as new information is available, but no later than 21:00 (UTC+1).

The platform is generally accessible. Occasional slow load times and load errors are possible.
We are continuing to investigate the issue related to the performance of the AskCody platform.

We will continue to update as soon as new information is available, but no later than 19:00 (UTC+1).

The platform is currently slow to load, though generally accessible. Occasional load errors are possible.
We are continuing to investigate the issue related to the performance of the AskCody platform.

We will continue to update as soon as new information is available, but no later than 17:00 (UTC+1).

The platform is currently slow to load, though generally accessible. Occasional load errors are possible.
We are continuing to investigate the issue related to the performance of the AskCody platform.

We will continue to update as soon as new information is available, but no later than 15:00 (UTC+1).

The platform is currently accessible. Slow loading and load errors may still be encountered.
We are continuing to investigate the issue related to the performance of the AskCody platform.

We will continue to update as soon as new information is available, but no later than 13:30 (UTC+1).

The platform is generally very slow to access with occasional load errors possible.
We are continuing to investigate the issue related to the performance of the AskCody platform.

We will continue to update as soon as new information is available, but no later than 12:00 (UTC+1).

The platform is generally inaccessible with only occasional access possible.
We are continuing to investigate the issue related to the performance of the AskCody platform.

We will continue to update as soon as new information is available, but no later than 10:30 (UTC+1).

We are continuing to investigate the issue related to the performance of the AskCody platform.
The platform is currently accessible. Occasional slow loading or load errors may be encountered.

We will continue to update as soon as new information is available, but no later than 10:00 (UTC+1).

We are continuing to investigate the issue related to the performance of the AskCody platform.
The platform is currently accessible. Occasional slow loading or load errors may be encountered.

We will continue to update as soon as new information is available, but no later than 08:30 (UTC+1).

We are continuing to investigate the issue related to the performance of the AskCody platform.
The platform is currently accessible. Occasional slow loading or load errors may be encountered.

We will continue to update as soon as new information is available, but no later than 07:15 (UTC+1).

The platform is currently accessible. Occasional slow loading or load errors may be encountered.
We are continuing to investigate the issue related to the performance of the AskCody platform.

We will continue to update as soon as new information is available, but no later than 21:00 (UTC+1).

We are continuing to investigate the issue related to the performance of the AskCody platform.
The platform is currently accessible; however, slow loading and few load errors may be encountered.

We will continue to update as soon as new information is available, but no later than 19:00 (UTC+1).

We are continuing to investigate the issue related to the performance of the AskCody platform.
The platform is currently accessible; however, slow loading and load errors may be encountered.

We will continue to update as soon as new information is available, but no later than 17:00 (UTC+1).

The platform is currently accessible. Slow loading and load errors may be encountered.
We are continuing to investigate the issue related to the performance of the AskCody platform.

We will continue to update as soon as new information is available, but no later than 15:30 (UTC+1).

We are continuing to investigate the issue related to the performance of the AskCody platform.
The platform is currently accessible. Slow loading and load errors may be encountered.

We will continue to update as soon as new information is available, but no later than 14:00 (UTC+1).

We are continuing to investigate the issue related to the performance of the AskCody platform.
The platform is currently accessible. Slow loading and load errors may be encountered.

We will continue to update as soon as new information is available, but no later than 12:30 (UTC+1).

Dearest customer,
We are aware of the pain and frustration the platform challenges are causing right now. Our team has been working since Thursday on solving the issues and we are doing everything we can to get it all up and running again as fast as possible.
We will continue to provide you with running updates, to let you know that we are actively working on solving these issues and to provide you with insight as the situation develops.
Once everything is restored, we will continue to be alert and monitor the platform and we will deliver a Postmortem to explain everything after resolution.
We apologise for the pain this causes you.

We are continuing to investigate the issue related to the performance of the AskCody platform.
The platform is generally unavailable.

We will continue to update as soon as new information is available, but no later than 11:00 (UTC+1).

We are continuing to investigate the issue related to the performance of the AskCody platform.
When using the platform you will experience slow load times and possible timeouts.

We will continue to update as soon as new information is available, but no later than 09:30 (UTC+1).

We are continuing to investigate the issue and working on a hypothesis currently.
The impact of these efforts is expected to be conclusive on Monday.

We will continue with regular updates on the morning of 23-01-23. Before then, in case of any outages on the platform, we will make sure to update you.

Load times have varied since the last update, but they are not back to normal levels.
We are continuing to investigate the issue.

We will continue to update as soon as new information is available, but no later than 18:00 (UTC+1).

We are continuing to investigate the issue.
The platform is currently available; however, load times can be slow.

We will continue to update as soon as new information is available, but no later than 15:30 (UTC+1).

We are continuing to investigate the issue.
The platform is currently available; however, load times can be slow.

- The "Slow load times" incident created this morning was determined to relate to this one and will be merged from here.

We will continue to update as soon as new information is available, but no later than 14:00 (UTC+1).

We are continuing to investigate the issue.
The platform is currently available; however, load times can be slow.

We will continue to update as soon as new information is available, but no later than 12:30 (UTC+1).

We are continuing to investigate the issue.
The platform is currently available; however, load times are slow.

We will continue to update as soon as new information is available, but no later than 11:30 (UTC+1).

The AskCody platform is experiencing a major outage.
We are currently investigating the issue.

We will continue to update as soon as new information is available, but no later than 10:30 (UTC+1)

We are currently experiencing an increase in load times across the EU side of the AskCody platform.

Normal operation has been restored for the platform and all affected components.
We will continue working to confirm causes as well as monitor performance over the coming days.

We are continuing the investigation of the issue.
We will provide the next update before 15:05 (UTC+1).

While we continuously get more elements of the issue confirmed, we are not locked down on a single cause. We are keeping the investigation open to uncover the root cause and get the issue resolved.
We will provide the next update before 14:05 (UTC+1)

Work to mitigate the issue as well as determine the cause and resolution is ongoing.
Refreshing the portal page can, for some, grant sporadic access to the portal, but we still deem this a Major Outage.
The success rate is up for loading the add-ins, but it is not on target. Displays and Dashboards should show calendar events, but on-screen interactions have errors. Check-in Kiosk should load, but have errors for check-ins.
We will provide the next update before 13:05 (UTC+1)

While some operation has been restored and sporadic acces is possible, this is still a major incident to us and work to determine the cause and resolution is ongoing.
We will provide the next update before 12:05 (UTC+1)

We are continuing to investigate the issue. We will strive to provide hourly updates on the status of the major outage.

We are working on a lead to what we believe to be the cause of major outage across the platform. We are, however, still investigating the issue. The outage is affecting both the AskCody Management Portal, as well as the Add-ins.

A major outage across the AskCody platform is currently being investigated.