Datto SaaS Protection - Degraded M365 Exchange Service Backups

We are continuing to monitor for any further issues.

We continue to see M365 Exchange backup success rates improve across the fleet and continue to monitor the backlog to ensure re-ingest of M365 Exchange services complete.


Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :

https://saasprotection.datto.com/help/M365/Content/Troubleshooting/errors/error-exchangeserverbusyerror-401-errorserverbusy.html?Highlight=throttling


Please note : Limit increases are temporary and will need to be applied again as needed.

We continue to see improvements in Exchange backup success rates and will continue to monitor progress.


Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :

https://saasprotection.datto.com/help/M365/Content/Troubleshooting/errors/error-exchangeserverbusyerror-401-errorserverbusy.html?Highlight=throttling


Please note : Limit increases are temporary and will need to be applied again as needed.


Subscribe to Datto SaaS Protection on the Status Page for up to date information.


You can monitor the current status of this issue at https://status.datto.com/

M365 Exchange backup success rates continue to improve since Microsoft completed the process to pause the reindexing of M365 Exchange mailboxes on Saturday, April 20th.


Our team continues to monitor M365 Exchange backup success rates as we continue to re-ingest this backlog over the coming days.


Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :

https://saasprotection.datto.com/help/M365/Content/Troubleshooting/errors/error-exchangeserverbusyerror-401-errorserverbusy.html?Highlight=throttling


Please note : Limit increases are temporary and will need to be applied again as needed.

We continue to see an improvement in the M365 Exchange backup success rates since Microsoft completed the process to pause reindexing Exchange mailboxes on Saturday, April 20th.


As this backlog is re-ingested over the coming days, the backup success rates will continue to improve.


Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :

https://saasprotection.datto.com/help/M365/Content/Troubleshooting/errors/error-exchangeserverbusyerror-401-errorserverbusy.html?Highlight=throttling


Please note : Limit increases are temporary and will need to be applied again as needed.

On Saturday, April 20th, Microsoft completed the process to pause reindexing Exchange mailboxes – the root cause of this issue - for all Datto SaaS Protection and Backupify customers.


Since the reindexing was paused, we have seen an improvement in the Exchange backup success rate, as the backlog of re-indexed data is re-ingested. As this backlog is re-ingested over the coming days, the backup success rates will continue to improve.


Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :

https://saasprotection.datto.com/help/M365/Content/Troubleshooting/errors/error-exchangeserverbusyerror-401-errorserverbusy.html?Highlight=throttling

Please note : Limit increases are temporary and will need to be applied again as needed.

Microsoft has agreed to pause the re-indexing process for all impacted Datto SaaS Protection and Backupify domains. As the SaaS Protection and Backupify products work through the backlog of data that has already been re-indexed, the backup success rates will improve.


The Datto R&D team is continuing to work on mitigations to process through this backlog at a faster rate.


Some domains may still be throttled while the system works through the re-index backlog. The mitigation below can be applied to any domains that are still seeing Exchange backup failures as those domains work through the backlog.


Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :
https://saasprotection.datto.com/help/M365/Content/Troubleshooting/errors/error-exchangeserverbusyerror-401-errorserverbusy.html?Highlight=throttling


Please note : Limit increases are temporary and will need to be applied again as needed.

Microsoft has confirmed that “significant changes” to their Exchange APIs involved re-indexing certain folders, causing all items in those folders to be re-synchronized and re-ingested by Datto SaaS Protection and Backupify.


Microsoft has stated that this issue is impacting multiple backup vendors.


This issue has resulted in increased traffic to the Microsoft API and subsequently increased throttling of requests, which caused a corresponding reduction in backup success rates, particularly for Microsoft Exchange mailboxes.


The queue of data that must be re-ingested is decreasing at a linear rate and customers will see their Exchange backup success rate improve as their re-ingest completes.


We are working directly with Microsoft’s Messaging Product team on a 24x7 basis to reduce the impact of this issue and it has been escalated to a ‘Sev A’ – Microsoft’s top level.


We are also rolling out additional infrastructure to assist in processing the additional data being sent from Microsoft.


Select nodes are being moved to 1 backup per day in an attempt to further reduce the impact of this issue.


Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :

https://saasprotection.datto.com/help/M365/Content/Troubleshooting/errors/error-exchangeserverbusyerror-401-errorserverbusy.html?Highlight=throttling

Please note : Limit increases are temporary and will need to be applied again as needed.

This issue is also impacting some users where backups will start and not complete. A fix for the incomplete backups issue has been identified and is being rolled out incrementally beginning Thursday, April 18 and is expected to complete by April 24th.

Microsoft has confirmed that a change to the Microsoft API has caused data to re-ingest for a sub-set of folders within certain users’ Exchange mailboxes and is impacting multiple backup vendors.


This has resulted in increased traffic to the Microsoft API and subsequently increased throttling of requests, particularly for Microsoft Exchange mailboxes.


As of Tuesday, April 16th, this issue was escalated within Microsoft to a SEV A and the Datto R&D team is working directly with Microsoft engineers on a 24x7 basis to bring the issue to resolution.


In parallel, additional API throttling mitigation changes were deployed on April 16th and backup success rates for both Microsoft Teams and SharePoint backups have now returned to normal levels.


Select nodes are being moved to 1 backup per day in an attempt to further reduce the impact of this issue.


Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :
https://saasprotection.datto.com/help/M365/Content/Troubleshooting/errors/error-exchangeserverbusyerror-401-errorserverbusy.html?Highlight=throttling


Please note : Limit increases are temporary and will need to be applied again as needed.


This issue is also impacting some users where backups will start and not complete. A fix for the incomplete backups issue has been identified and is being rolled out incrementally beginning Thursday, April 18 and is expected to complete by April 24th.

An unexpected change to the Microsoft API has resulted in Datto SaaS Protection and Backupify re-ingesting data for a sub-set of folders within certain users’ Exchange mailboxes. This has resulted in increased traffic to the Microsoft API and subsequently increased throttling of requests.


This issue has been escalated within Microsoft to a SEV A and the Datto R&D team is working directly with Microsoft engineers on a 24x7 basis to address the issue.


The platform continues to work through the queue of re-ingested data. We anticipate that the API throttling will decrease as the queue of re-ingested data is processed.


In parallel, additional API throttling mitigation changes were deployed on April 16th to improve backup success rates. This change is showing significant improvement in both Microsoft Teams and SharePoint backups, which are expected to return to normal by end of day April 16th.


Select nodes are being moved to 1 backup per day in an attempt to further reduce the impact of this issue.


Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :
https://saasprotection.datto.com/help/M365/Content/Troubleshooting/errors/error-exchangeserverbusyerror-401-errorserverbusy.html?Highlight=throttling


Please note : Limit increases are temporary and will need to be applied again as needed.


This issue is also impacting some users where backups will start and not complete. A fix for the incomplete backups issue has been identified and is being rolled out incrementally beginning Wednesday, April 17 and is expected to complete by April 24th.

An unexpected change to the Microsoft API has resulted in Datto SaaS Protection and Backupify re-ingesting data for a sub-set of folders within certain users’ Exchange mailboxes. This has resulted in increased traffic to the Microsoft API and subsequently increased throttling of requests.


Over the weekend, the platform continued to work through the queue of re-ingested data. We anticipate that the API throttling will decrease as the queue of re-ingested data is processed.


Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :

https://saasprotection.datto.com/help/M365/Content/Troubleshooting/errors/error-exchangeserverbusyerror-401-errorserverbusy.html?Highlight=throttling


Please note : Limit increases are temporary and will need to be applied again as needed.


This issue is also impacting some V2 users where backups will start and not complete. A fix for the incomplete backups issue on V2 has been identified and is being rolled out beginning Monday, April 15 and the rollout is expected to complete by mid-week.

An unexpected change to the Microsoft API has resulted in Datto SaaS Protection and Backupify re-ingesting data for a sub-set of folders within certain users’ Exchange mailboxes. This has resulted in increased traffic to the Microsoft API and subsequently increased throttling of requests.


The impact is a reduction in backup success rates for the users experiencing this issue.


We are actively working with Microsoft to resolve this problem.


Mitigation :
The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :

https://saasprotection.datto.com/help/M365/Content/Troubleshooting/errors/error-exchangeserverbusyerror-401-errorserverbusy.html?Highlight=throttling

Please note : Limit increases are temporary and will need to be applied again as needed.

Subscribe to Datto SaaS Protection on the Status Page for up to date information.

You can monitor the current status of this issue at https://status.datto.com/

We are currently aware of a problem where a subset of Datto SaaS Protection users can experience a M365 Exchange failure due to error "ExchangeServerBusyError".


Mitigation : The steps outlined in the following KB will assist in mitigating this issue by temporarily increasing the Microsoft throttling limit :


https://saasprotection.datto.com/help/M365/Content/Troubleshooting/errors/error-exchangeserverbusyerror-401-errorserverbusy.html?Highlight=throttling


Please note : Limit increases are temporary and will need to be applied again as needed.


Our Engineering team is currently investigating the problem and working with Microsoft towards a resolution.


Subscribe to Datto SaaS Protection on the Status Page for up to date information.


You can monitor the current status of this issue at https://status.datto.com/