[HSE20260810-01-I] Incident: Degraded Performance: 365 Total Backup Service on CA Region

The 365 Total Backup Service is only available with a degraded performance for a small subset of tenants on the Canada Region.

The cause is currently being analyzed in more detail.

The 365 Total Backup Service is unaffected on other Regions.
Other Services and Email Flow are also NOT affected.

Our technical team has successfully identified the root cause of the 365TB issue affecting the CA Region. Our team is working around the clock to implement a permanent, sustainable fix.

We appreciate your continued patience and understanding as we work to restore full system stability. We will be updating this space as soon as new information becomes available.

Our Engineering and Operations Teams are still actively focused on implementing a definitive solution for the CA Region.

In the meantime, we are currently also performing some tests and calculations in order to work out an ETA, which we can share.

Our Technical Team has just finished developing a fix for the 365TB issues on the CA Region.

This fix is currently being tested by our QA Team and will be deployed as soon as it gets the greenlight. If everything goes well, we will be aiming to rollout this fix by tomorrow.

Our engineering team wrapped up the initial fix, but we ran into a possible issue during QA testing. To prevent any downside or risk, we need to run a few more tests and validate everything properly.

The release will therefore be pushed back slightly to allow for extended automated testing throughout the weekend, with the rollout now expected early next week.

We appreciate your patience as we make sure this fix is thorough before it goes live.

Our Technical Team is scheduled to deploy the fix tomorrow morning, pending the final automated tests running this evening.

Following the deployment, systems are expected to begin a gradual recovery as early as tomorrow afternoon.

Performance will be monitored closely, and further updates will be provided as operations return to normal. Thank you for your continued patience.

Our Technical Team has successfully deployed the fix on the CA Region. Systems are recovering at a steady pace, with 75% of the affected tenants already back to normal.

Performance will continue to be monitored closely, and we will share a final update once systems are fully recovered. We appreciate your understanding throughout this process.

Our Technical Team reports that system recovery is continuing to progress at a gradual pace.
We will continue to closely oversee the performance and provide updates as recovery continues.

The final phase of system recovery is taking longer than initially anticipated, however we are seeing steady progress and are estimating a full recovery by mid-next week.

Performance is being closely monitored as we work through these final steps. We apologise for the inconvenience such delays may have caused, and we will make sure to communicate any updates as we get them.

The final phase of system recovery is progressing as planned.

System performance will continue to be closely monitored throughout the weekend, with further updates provided as they become available.

Thank you for your continued patience and understanding.

Weekend monitoring identified issues that slowed recovery and required the process to be restarted using improved tools. Recovery is expected to continue throughout the week, with completion anticipated toward the end of the week.

We will continue to monitor progress and provide timely updates in the coming days as the service gradually improves for tenants in the CA region.

We apologize for the delay. Our Product and Systems Engineering teams remain fully engaged and are working to restore all systems as efficiently as possible.

Restoration using the improved tools is progressing steadily and expected timelines remain unchanged, and a substantial number of tenants has already been successfully recovered.

Our Engineering teams remain fully focused on completing a stable and thorough recovery and we will continue to provide timely updates throughout the week.

Thank you for your continued cooperation and patience.