Elevated 504 responses from Exchange Server

This incident has been resolved.

We have finished the deployment of the fix to our Sync component in our production environment. We are closely monitoring the results.

We have finished the deployment of the fix to our API component in our production environment. We are closely monitoring the results.
We are now starting the deployment of the fix to the Sync component in our production environment.

The deployment is 75% done and we are already seeing an improvement in the error rate on /send. We will confirm when the deployment is completed.

The smoke tests on our staging environment were successful. We are now starting the deployment of the aforementioned fix to our production environment.

The deployment has made it to our staging environment. We are running our smoke tests there before deploying to production.

We want to give visibility on an advisory issue that Microsoft posted earlier today that we believe is related to the current incident: https://admin.microsoft.com/adminportal/home#/healthoverview/:/alerts/EX540990 (Microsoft 365 Admin account required). We are seeing error levels decreasing since 1PM EST confirming they are rolling back that change. Please note we are still proceeding with the deployment on our side in an effort to speed up resolution for all users.

We are still preparing our deployment by running our end-to-end test suite. We'll start the deployment once that's done. Thank you for your continued patience.

The fix is implemented and tested but not deployed yet. We are preparing the deployment. We will keep you posted.

We believe we have identified the issue and are implementing a fix.

We are following a lead on what the root cause might be, however we are still investigating.

Please ask your customers’ mail administrators for assistance to raise tickets with Microsoft to help identify the cause of the increase in Fanout errors. This Fanout error suggests Microsoft have an issue with their storage, this is likely a transitory error with Microsoft ongoing since 11/04 18:00 UTC

An internal server error occurred. The operation failed., Fanout timed out

This is impacting a small percentage of Office 365 accounts. From past experience we have produced the following KB article: https://support.nylas.com/hc/en-us/articles/4505725124253

Please ask your customers’ mail administrators to raise tickets with Microsoft to place pressure on them to identify the cause of the increase in Fanout errors. This Fanout error suggests Microsoft have an issue with their storage, this is likely a transitory error with Microsoft ongoing since 11/04 18:00 UTC

An internal server error occurred. The operation failed., Fanout timed out

This is impacting a small percentage of Office 365 accounts. From past experience we have produced the following KB article: https://support.nylas.com/hc/en-us/articles/4505725124253

We have confirmed that this is impacting a small percentage of our customers' users on EWS Exchange.

We're still investigating this issue.

Thank you for your continued patience.

We are aware of elevated 504 error responses when sending from the Exchange server and are currently investigating this issue.