Increased error rate communicating with Outlook.com calendars

We've now worked around the API error such that our Outlook.com calendar sync performance is no longer affected, and we are therefore marking this incident as resolved. The previous caveat around having an outdated view of affected users' secondary calendars no longer applies, due to an extra mitigation put in place yesterday and confirmed with successful Telemetry results overnight.

We have seen no noticeable change in Microsoft's API behaviour, with a similar error rate being seen all week.

The situation remains the same; Due to our mitigations, calendar sync is working for all users, and the only issue affecting users as a result of this incident in practice is as previously mentioned:

- We are unable to pick up changes to affected users' calendar lists and may have an outdated view of secondary calendars as a result.

We've deployed a mitigation which allows for users connecting Outlook.com accounts for the first time (or who had connected for the first time earlier during this incident) to begin syncing successfully.

We've still not seen acknowledgement or action from Microsoft of the underlying issue.

Due to our mitigations, calendar sync is working for all users, and the only issue affecting users as a result of this incident in practice is as previously mentioned:

- We are unable to pick up changes to affected users' calendar lists and may have an outdated view of secondary calendars as a result.

There has been no significant change in behaviour seen on Microsoft's API, but we are testing a change to our calendar listing mechanism which may allow us to completely work around Microsoft's erroring behaviour, which would resolve this issue completely.

We will update as we have progress, for now the scenario is the same as the previous update:

- Calendar event syncing is working for Outlook.com calendars we were previously aware of.
- We are unable to pick up changes to affected users' calendar lists and may have an outdated view of secondary calendars as a result.
- Users connecting Outlook.com calendar accounts to Cronofy for the first time may have issues getting successfully connected.

Our workaround has been successful, and has allowed calendar sync to resume for the affected users.

We continue to observe the elevated error rate on Microsoft's API, and will not be able to fully resolve the issue until this underlying cause is fixed.

During this time:
We are unable to pick up changes to affected users' calendar lists and may have an outdated view of secondary calendars as a result.
Users connecting Outlook.com calendar accounts to Cronofy for the first time may have issues getting successfully connected.

While Microsoft hasn't acknowledged the issue yet, this has been raised by other integrators, for example:
https://learn.microsoft.com/en-us/answers/questions/2279133/getting-500-response-errors-to-get-me-calendars-al
https://github.com/microsoftgraph/microsoft-graph-explorer-v4/issues/3861

We will continue to monitor, and shall update when there are notable changes or at least by 9AM UTC tomorrow, June 3rd.

We've now fully rolled out our workaround and are monitoring the results as we work through our backlog of earlier failures.

Since we observed errors specifically when attempting to list calendars, as a temporary measure we are allowing the rest of the sync process to continue based on our latest copy of the user's calendar list. This allows calendar event data to resume syncing and avoids us having stale availability information for the affected users.

This is not a full resolution since it does mean that if affected users create or edit secondary calendars, we won't pick up the newly added calendars or new calendar names.

We've found that we are seeing errors only from a particular API call which we use to list calendars, and have seen success with a limited rollout of a workaround that allows the rest of the sync process to continue. We are watching its progress and increasing our confidence in order to roll this out further.

We've found that approximately one third of Outlook.com calendars are being affected. We do not believe a change our side has caused the elevated errors, but nor has Microsoft acknowledged an incident on their side.

We are continuing to investigate the affected cohort, and which API calls are affected, in case there is any workaround we can implement on our side.

We've identified that a cohort of Outlook.com profiles are seeing an increased error rate when we communicate with Microsoft's APIs, causing sync failures for those affected.

We're investigating the root cause and will update this incident as we understand more.