Microsoft Teams Issues

Microsoft has made serveral optimizations and mitigation effors and have confirmed that the issues with Microsoft Teams services and features has been resolved.

***CallTower will be closing this master case and all associated cases. If provided by Microsoft an Official Root Cause Analysis (RCA) will be available and provided to customers upon request.

Title: Some users may experience multiple issues with their Microsoft Teams

User impact: Users may experience multiple issues with their Microsoft Teams.

More info: Affected scenarios included, but weren’t limited to the following:

- Users logging in or unlocking their devices after some time may have had missing messages.
- Users may have been unable to load messages in channels and chats.
- Users were unable to view or download their media (images, videos, audio, call recordings, code snippets).
- Some users were experiencing delays sending and receiving chat messages.
- Anonymous users may have been unable to join meetings.
- Teams connectors for Power Automate/Power Apps were experiencing errors.
-Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online.
- Users may have been unable to load previous Copilot history, or new history was not written
- Bots may have been unable to download attachments
- Sending and receiving read receipt notifications may have been delayed
- Users may have experienced issues resuming or rejoining meetings.
- Users within organizations leveraging an Information Barrier may have experienced issues joining meetings
- Users performing a cold boot may have been unable to log into teams and would have seen an "oops" page

Final status: After extended monitoring and various optimizations and mitigation efforts, we’ve confirmed that our Microsoft Teams service and features have been restored or returned to optimal health.

Scope of impact: This issue could have potentially impacted any Microsoft Teams user in the scenarios outlined in the More info section.

Start time: Friday, January 26, 2024 at 7:55 AM MST

End time: Friday, January 26, 2024 at 6:30 PM MST

CallTower continues to monitor the Microsoft issue affecting some of our customers and have seen significant improvements. We are not experiencing any current impairments. Microsoft has provided an update regarding the issue and the affected services.

Latest Microsoft Update: Jan 26, 2024, 5:22 PM MST

Title: Some users may experience multiple issues with their Microsoft Teams

User impact: Users may experience multiple issues with their Microsoft Teams.

More info: Affected scenarios include, but aren't limited to the following:

Mitigated/improving:

- [Updated] Users logging in or unlocking their devices after some time may have missing messages. We believe that this symptom has been addressed.
- [Updated] Users may have been unable to load messages in channels and chats. We believe that this symptom has been addressed.
- [Updated] Users were unable to view or download their media (images, videos, audio, call recordings, code snippets). We believe that this symptom has been addressed.
- [Updated] Some users were experiencing delays sending and receiving chat messages. We believe this symptom has been largely addressed, though some users may still experience brief delays.
- [Updated] Anonymous users may be unable to join meetings. We’re seeing improvement in this area but some users may still be affected by this scenario.
- [Updated] Teams connectors for Power Automate/Power Apps were experiencing errors. We believe that this symptom has been addressed.
- [Updated] Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online. This symptom has been addressed.

Still affected/undergoing further evaluation:

- Users may be unable to load previous Copilot history, or new history is not written
- Bots may be unable to download attachments
- Sending and receiving read receipt notifications may be delayed
- Users may experience issues resuming or rejoining meetings.
- Users within organizations leveraging an Information Barrier may experience issues joining meetings
- Users performing a cold boot may not be able to log into teams and will see an "oops" page

Current status: We’re seeing significant improvements with many of the Teams features affected by this incident as we continue to apply mitigations and optimize traffic. We’re evaluating the remaining impact scenarios and are developing fixes and workstreams to address all lingering impact associated with this event. Please see the “more info” section above for the latest information on each impact scenario.

We understand the impact an issue like this can have on your organization, and we appreciate your partnership and patience as we work to remediate this issue.

Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section.

Start time: Friday, January 26, 2024 at 7:55 AM MST

***CallTower will provide the next update by Saturday, January 27, 2024 at 10:00 AM MST, unless the issue is resolved by Microsoft.

CallTower continues to monitor the Microsoft issue affecting some of our customers. We have seen major improvements in our testing, and aren’t seeing any major impairment currently. Microsoft has provided an update regarding the issue and the affected services.

Latest Microsoft Update: Jan 26, 2024, 3:58 PM MST

Title: Some users may experience multiple issues with their Microsoft Teams
User impact: Users may experience multiple issues with their Microsoft Teams.
More info: Affected scenarios include, but aren't limited to:
- Anonymous users may be unable to join calls or meetings
- Users within organizations leveraging an Information Barrier may experience issues joining meetings
- Users performing a cold boot may not able to log into teams and will see an "oops" page
- Users logging in or unlocking their devices after some time may see missing messages
- Users may fail to load messages in channels and chats
- Users are unable to view or download their media (images, videos, audio, call recordings, code snippets)
- Some messages may experience delays being sent
- Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online
- Users may be unable to load previous Copilot history, or new history is not written
- Bots may be unable to download attachments
- Sending and receiving read receipt notifications may be delayed
- Anonymous users may be unable to join meetings
- Teams connectors for Power Automate/Power Apps may experiencing errors

Current status: We're continuing to apply mitigations across the affected infrastructure and our telemetry is showing additional improvement in the user experience, though many customers are still affected by this issue. We're also working to apply fixes to address individual affected Teams features in parallel while our broader remediation strategy is ongoing. We're evaluating any and all additional workstreams that will allow us to reduce the impact to those customers that are still affected.
We understand the impact an issue like this can have on your organization, and we appreciate your partnership and patience as we work to remediate this issue.
Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section.
Start time: Friday, January 26, 2024 at 7:55 AM MST
.
Next update by: Friday, January 26, 2024 at 5:30 PM MST

CallTower continues to monitor the Microsoft issue affecting some of our customers. Microsoft has provided an update regarding the issue and the affected services.

Latest Microsoft Update: Jan 26, 2024, 3:02 PM MST

Title: Some users may experience multiple issues with their Microsoft Teams

User impact: Users may experience multiple issues with their Microsoft Teams.

More info: Affected scenarios include, but aren't limited to:

- Users performing a cold boot may not able to log into teams and will see an "oops" page
- Users logging in or unlocking their devices after some time may see missing messages
- Users may fail to load messages in channels and chats
- Users are unable to view or download their media (images, videos, audio, call recordings, code snippets)
- Some messages may experience delays being sent
- Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online
- Users may be unable to load previous Copilot history, or new history is not written
- Bots may be unable to download attachments
- Sending and receiving read receipt notifications may be delayed
- Anonymous users may be unable to join meetings
- Teams connectors for Power Automate/Power Apps may experiencing errors


Current status: Our network and backend service optimization efforts are ongoing and we're monitoring internal telemetry to confirm that our efforts are effectively reducing the impact to customers. We're continuing to scale up our services and identify new workstreams intended to remediate this problem more rapidly. Although many customers remain impacted by this issue, we're seeing a reduction in errors and an increase in availability.

We understand the impact an issue like this can have on your organization, and we appreciate your partnership and patience as we work to remediate this issue.

Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section.

Start time: Friday, January 26, 2024 at 7:55 AM MST
.

Next update by: Friday, January 26, 2024 at 4:00 PM MST

Microsoft and some of our customers have reported service issues associated with Teams. Microsoft has provided an update regarding the issue and the affected services.

Latest Microsoft Update:
Jan 26, 2024, 1:02 PM MST

Title: Some users may experience multiple issues with their Microsoft Teams

User impact: Users may experience multiple issues with their Microsoft Teams.

More info: Affected scenarios include, but aren't limited to:

- Users performing a cold boot may not able to log into teams and will see an "oops" page
- Users logging in or unlocking their devices after some time may see missing messages
- Users may fail to load messages in channels and chats
- Users are unable to view or download their media (images, videos, audio, call recordings, code snippets)
- Some messages may experience delays being sent
- Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online
- Users may be unable to load previous Copilot history, or new history is not written
- Bots may be unable to download attachments
- Sending and receiving read receipt notifications may be delayed
- Anonymous users may be unable to join meetings
- Teams connectors for Power Automate/Power Apps may experiencing errors

Current status: Our failover operation did not provide the anticipated relief to end users
in North and South America regions, and we are now working to optimize traffic patterns as part of the mitigation effort. We're applying configuration changes across the affected network infrastructure to reduce customer impact as quickly as possible.

Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section.

Start time: Friday, January 26, 2024 at 7:55 AM MST

Next update by: Friday, January 26, 2024 at 2:00 PM MST

*CallTower has opened a master case to investigate this issue as it has been reported by more than one CallTower customer. This issue may or may not directly affect your services and is currently under investigation. If you are affected, please report the issue to support@calltower.com. CallTower intends to post updates on status.calltower.com approximately every 30 minutes, unless identified otherwise. This issue is being investigated as a top priority.

CallTower is investigating reports of numerous issues affecting Microsoft Teams. Microsoft has acknowledged an issue in their status TM710344. We are closely monitoring the issue and will continue to provide updates as we receive them.

Most recent Microsoft update:

TM710344, Last updated: Jan 26, 2024, 11:16 AM MST
Estimated start time: Jan 26, 2024, 8:37 AM MST
Affected services
Microsoft Teams

Title: Some users may experience multiple issues with their Microsoft Teams

User impact: Users may experience multiple issues with their Microsoft Teams.

More info: Affected scenarios include, but aren't limited to:
- Users performing a cold boot may not able to log into teams and will see an "oops" page
- Users logging in or unlocking their devices after some time may see missing messages
- Users may fail to load messages in channels and chats
- Users are unable to view or download their media (images, videos, audio, call recordings, code snippets)
- Some messages may experience delays being sent
- Call Recordings might take longer to appear in user's OneDrive for Business and SharePoint Online
- Users may be unable to load previous Copilot history, or new history is not written
- Bots may be unable to download attachments
- Sending and receiving read receipt notifications may be delayed
- Anonymous users may be unable to join meetings
- Teams connectors for Power Automate/Power Apps may experiencing errors

Current status: We've completed the failover in the Europe, Middle East, and Africa (EMEA) region and telemetry is showing improvement. We're continuing with the failover processes in the Americas, and are monitoring whilst these processes complete.

Scope of impact: This issue can potentially impact any Microsoft Teams user in the scenarios outlined in the More info section.

Start time: Friday, January 26, 2024 at 7:55 AM MST

Next update by: Friday, January 26, 2024 at 1:00 PM MST