Users in Africa may be unable to access multiple Microsoft 365 services or features

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access multiple Microsoft 365 services or features.

Current status: We’ve added additional capacity and can confirm the service has remained healthy during the start of core business hours. We’re continuing to closely monitor service telemetry to ensure the service remains stable.

Scope of impact: The issue primarily impacts users within the Africa region. However, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Start time: Thursday, March 14, 2024, at 10:30 AM UTC

Preliminary root cause: Multiple major fiber cables which manage requests in and out of the Africa region are broken.

Next update by: Monday, March 18, 2024, at 11:00 PM UTC

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access multiple Microsoft 365 services or features.

Current status: Our telemetry indicates that application performance remains stable for services, as load-balancing efforts continue. We’ll be working over the weekend to source more capacity in preparation for higher traffic as core business hours resume on Monday, March 18, 2024.

Additionally, we've re-enabled Azure Front Door (AFD) routing within the South Africa region and are continuing to monitor for any disruptions in service, though we are observing stable performance.

Scope of impact: The issue primarily impacts users within the Africa region. However, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Start time: Thursday, March 14, 2024, at 10:30 AM UTC

Preliminary root cause: Multiple major fiber cables which manage requests in and out of the Africa region are broken.

Next update by: Monday, March 18, 2024, at 9:00 AM UTC

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access multiple Microsoft 365 services or features.

Current status: We've added the additional network capacity, however, there has been a slight amount of instability in the newly added links. We're working with the partner carriers who manage the fiber cables to investigate the issues. Additionally, we're planning to add additional fiber links within the region over the weekend, and then further capacity into next week.

We're in the process of removing some background throttling that was initially applied to the Exchange Online services to reduce service load. Once our additional capacity is stable, we plan to re-enable Azure Front Door (AFD) routing within the South Africa region to improve some application performance.

Scope of impact: The issue primarily impacts users within the Africa region. However, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Start time: Thursday, March 14, 2024, at 10:30 AM UTC

Preliminary root cause: Multiple major fiber cables which manage requests in and out of the Africa region are broken.

Next update by: Friday, March 15, 2024, at 11:00 PM UTC

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access multiple Microsoft 365 services or features.

Current status: We're continuing to resolve an issue causing network instability which is preventing us from completing an additional network capacity upgrade. We estimate this issue will be fixed by March 15, 2024, at 6:00 PM UTC. Once the issue is resolved, we'll continue to activate the network capacity upgrade, and, where possible, complete any additional traffic flow redirections to reduce the overall utilization of the affected infrastructure. In addition, we're continuing to monitor service telemetry to ensure Microsoft 365 services continue to function as expected.

Scope of impact: The issue primarily impacts users within the Africa region. However, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Start time: Thursday, March 14, 2024, at 10:30 AM UTC

Preliminary root cause: Multiple major fiber cables which manage requests in and out of the Africa region are broken.

Next update by: Friday, March 15, 2024, at 6:30 PM UTC

We're currently facing technical issues with activating an additional capacity upgrade due to network traffic instability. We're working to resolve the issue before introducing the additional capacity within the region to prevent further impact. We estimate the issue will be mitigated by March 15, 2024, at 6:00 PM UTC.

This quick update is designed to give the latest information on this issue.

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access multiple Microsoft 365 services or features.

Current status: We've completed an initial capacity upgrade and we're continuing our efforts to further increase the network capacity within the region to restore it to pre-incident levels. In parallel, we're reviewing background traffic data to complete more traffic flow redirections to continue reducing the overall utilization of the affected infrastructure. Overall network health within the region is steadily improving and reports from our telemetry indicate that the majority of Microsoft 365 services remain in a healthy state while these actions are in progress.

Scope of impact: The issue primarily impacts users within the Africa region. However, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Start time: Thursday, March 14, 2024, at 10:30 AM UTC

Preliminary root cause: Multiple major fiber cables which manage requests in and out of the Africa region are broken.

Next update by: Friday, March 15, 2024, at 3:00 PM UTC

We're continuing to redirect traffic flows to alternative paths to reduce the overall utilization of the affected infrastructure. Additionally, capacity within the region has been increased and we're working to further increase the overall capacity to pre-incident levels. We're continuing to closely monitor service telemetry to ensure our services remain in an optimal state while this work is in progress.

This quick update is designed to give the latest information on this issue.

We're continuing to perform multiple network optimizations to increase the overall capacity in the region. In parallel, we're closely monitoring our telemetry to ensure our services remain stable while this work is in progress.

This quick update is designed to give the latest information on this issue.

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access multiple Microsoft 365 services or features.

Current status: We have made progress in handling the fiber cable repairs to completely resolve the issue. In parallel, overall network health for the impacted region has been steadily improving. A majority of the affected Microsoft 365 services have returned to healthy levels. We continue to validate that our services remain at optimal levels while we implement additional capacity in the region. Based on our current progress, we expect to complete our capacity upgrades by March 15, 2024 at 2:00 PM UTC.

Scope of impact: The issue primarily impacts users within the Africa region. However, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Start time: Thursday, March 14, 2024, at 10:30 AM UTC

Preliminary root cause: Multiple major fiber cables which manage requests in and out of the Africa region are broken.

Next update by: Friday, March 15, 2024, at 1:00 PM UTC

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access multiple Microsoft 365 services or features.

Current status: We're continuing to address the delays that are preventing the fiber cables from being repaired. In parallel, our efforts to reroute traffic has been successful in minimizing impact that may be experienced during your business hours. We understand the impact this issue may have on your organization and we will continue to treat this incident with the highest priority and urgency.

Scope of impact: The issue primarily impacts users within the Africa region. However, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Start time: Thursday, March 14, 2024, at 10:30 AM UTC

Preliminary root cause: Multiple major fiber cables which manage requests in and out of the Africa region are broken.

Next update by: Friday, March 15, 2024, at 8:00 AM UTC

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access multiple Microsoft 365 services or features.

Current Status: Due to on-site complications the repair to the West Africa fiber cables has been delayed for at least 12 hours. This is preventing full resolution for this issue. There have been successful efforts to redirect traffic away from the region and on to healthy infrastructure, however we anticipate service disruptions as traffic peaks during core business hours in the region. We're continuing to closely monitor telemetry and investigating additional methods for optimizing available capacity.

Scope of impact: The issue primarily impacts users within the Africa region. However, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Start time: Thursday, March 14, 2024, at 10:30 AM UTC

Preliminary root cause: Multiple major fiber cables which manage requests in and out of the Africa region are broken.

Next update by: Friday, March 15, 2024, at 5:00 AM UTC

A transient dip in overall network capacity may have caused brief functionality issues for some services. We have since returned to stable levels and we're continuing to monitor the service and take proactive actions as needed. In parallel, we are implementing additional capacity while work to restore the West Africa fiber cables continues.

This quick update is designed to give the latest information on this issue.

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access multiple Microsoft 365 services or features.

Current status: Our analysis indicates that users should no longer be experiencing functionality issues, as traffic optimizations to provide additional capacity have been successful. However, we're encountering some delays in the effort to reconfigure the fibre cables in the West Africa region. We are now targeting Friday, March 15, 2024, at 2:00 AM UTC to complete this work and fully remediate impact to infrastructure in the region.

Scope of impact: The issue primarily impacts users within the Africa region. However, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Start time: Thursday, March 14, 2024, at 10:30 AM UTC

Preliminary root cause: Multiple major fibre cables which manage requests in and out of the Africa region are broken.

Next update by: Friday, March 15, 2024, at 3:00 AM UTC

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access multiple Microsoft 365 services or features.

Current status: We're observing steady progress in efforts to optimize traffic and re-route operations to healthy infrastructure. Many users will begin to see service functionality return to pre-impact levels. We're continuing to work on providing additional capacity to accommodate increased traffic levels when core business hours resume tomorrow across the Africa region.

We're still targeting Thursday, March 14, 2024, at 10:00 PM UTC to complete work on reconfiguring fibre cables in the West Africa region to provide the service with full processing throughput to completely remediate impact.

Scope of impact: The issue primarily impacts users within the Africa region. However, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Start time: Thursday, March 14, 2024, at 10:30 AM UTC

Preliminary root cause: Multiple major fibre cables which manage requests in and out of the Africa region are broken.

Next update by: Thursday, March 14, 2024, at 11:00 PM UTC

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access multiple Microsoft 365 services or features.

Current status: The major work to reconfigure the available fibre cables in the West Africa region is continuing to progress as expected. We still have a targeted ETA of Thursday, March 14, 2024, at 10:00 PM UTC to complete this work and provide the service with pre-impact levels of processing throughput, fully remediating impact.

In parallel, we're continuing to perform traffic optimizations and load-reducing operations on services within Exchange Online and Microsoft Teams to provide immediate short-term relief. As traffic levels in the Africa region continue to decrease in line with the end of core business hours, users will continue to experience incremental improvement.

Scope of impact: The issue primarily impacts users within the Africa region. However, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Start time: Thursday, March 14, 2024, at 10:30 AM UTC

Preliminary root cause: Multiple major fibre cables which manage requests in and out of the Africa region are broken.

Next update by: Thursday, March 14, 2024, at 10:30 PM UTC

We've triggered some additional load-reducing within the Exchange Online service to provide further relief.

This quick update is designed to give the latest information on this issue.

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access Multiple Microsoft 365 services or features.

Current status: We’re performed multiple actions on back-end processes to reduce service load and provide some relief. These actions were targeted to services including but not limited to; SharePoint Online, OneDrive for Business, Microsoft Teams and Azure Front Door routing. While we are still performing traffic shedding processing, a large focus is on reconfiguring and converting some available fibre links in the West Africa region into additional capacity to bring the overall throughput to pre-impact levels. Our current ETA for this action is 10:00 PM UTC. Furthermore, users will start to experiencing incremental improvement as business hours in the Africa region come to an end and traffic volumes reduce.

Scope of impact: The issue significantly impacts users within the Africa region, however, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Root cause: Multiple major fibre cables which manage requests in and out of the Africa region are broken.

Next update by: Thursday, March 14, 2024, at 6:00 PM UTC

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access Multiple Microsoft 365 services or features.

Current status: We continue to progress with the previously mentioned workstreams:
- The third-party carrier provider responsible for the fibre cables continues to address the underlying issue. They have provided an ETA of 12 hours for these actions.
- Microsoft 365 services are rerouting connections to alternate regions where possible.
- Microsoft Teams calling is routed through Europe and has provided some relief.
- SharePoint Online and OneDrive for Business have rerouted connections to an alternate data center within the Africa region, which has provided some relief.
- We’re performing work to reconfigure the available fibre cables in the Africa region to provide more processing throughput capabilities.

Scope of impact: The issue significantly impacts users within the Africa region, however, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Root cause: Multiple major fibre cables which manage requests in and out of the Africa region are broken.

Next update by: Thursday, March 14, 2024, at 4:00 PM UTC

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access Multiple Microsoft 365 services or features.

Current status: We’ve identified that multiple major fibre cables which manage requests in and out of the Africa region are broken. We’re undertaking multiple workstreams to address the issue:
- We’re working with the third-party carrier provider who manages the fibre cables to identify the cause and address the issue
- We’re redirecting multiple Microsoft 365 services and functionalities to leverage alternate healthy components where possible
- We’re performing traffic-management operations to optimize the usage of the available fibre cables and networking infrastructure within the region

Scope of impact: The issue significantly impacts users within the Africa region, however, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Next update by: Thursday, March 14, 2024, at 2:30 PM UTC

Title: Users in Africa may be unable to access multiple Microsoft 365 services or features

User impact: Users in Africa may be unable to access Multiple Microsoft 365 services or features.

Current status: We’ve identified a networking issue and we’re rerouting connections to alternate components to mitigate impact.

Scope of impact: The issue significantly impacts users within the Africa region, however, users in other geographic locations may be impacted if their connections are routed through impacted networking components.

Next update by: Thursday, March 14, 2024, at 1:30 PM UTC

Title: Investigating user reports of Microsoft 365 suite service issues

User impact: We don't have detail on impact at this time. We're seeing an increase in user reported issues above alerting threshold.

Current status: We are reviewing service telemetry and available information to determine if there is an issue happening and we'll update this message shortly with our latest findings.

This communication serves as a preliminary notification about a potential issue affecting your service.