Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.

More info: Some users may encounter delivery delays of up to approximately five hours. The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.

Current status: Our set of steps to reroute the free/busy CFMs are deploying to the affected environment and we expect to have an estimated time to resolution by the next scheduled update.

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.

Start time: Monday, July 17, 2023, at 3:00 PM UTC

Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.

Next update by: Tuesday, July 18, 2023, at 9:30 PM UTC

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.

More info: Some users may encounter delivery delays of up to approximately five hours. The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.

Current status: After further investigation, we're now working to implement a set of steps that will reroute the free/busy CFMs to the appropriate mailboxes.

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.

Start time: Monday, July 17, 2023, at 3:00 PM UTC

Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.

Next update by: Tuesday, July 18, 2023, at 7:30 PM UTC

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.

More info: Some users may encounter delivery delays of up to approximately five hours. The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.

Current status: We've disabled the free/busy publishing assistant in a small portion of affected infrastructure to test the efficacy of this mitigation and initial feedback indicates this hasn't had the positive effect we anticipated for reducing email delays. We're reviewing other potential options to mitigate impact while simultaneously assessing further long-term fix solutions.

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.

Start time: Monday, July 17, 2023, at 3:00 PM UTC

Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.

Next update by: Tuesday, July 18, 2023, at 5:30 PM UTC

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.

More info: Some users may encounter delivery delays of up to approximately four hours. The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.

Current status: We've deployed the revised fix but identified that this has only improved delays for some users and hasn't provided the expected relief for all users. We're assessing options of disabling the free/busy publishing assistant to prevent CFMs from being generated and causing email delivery delays. This will restore email delays but users may still see outdated free/busy information in Exchange Online. After disabling the assistant, we'll deploy a longer term fix to enable the assistant without causing an excessive load of CFMs.

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.

Start time: Monday, July 17, 2023, at 3:00 PM UTC

Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.

Next update by: Tuesday, July 18, 2023, at 3:30 PM UTC

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.

Current status: We've identified that the fix had an unintended impact and resulted in mail flow issues, generating NDRs for some users. We've reverted the fix that caused impact to mail flow and are now deploying a revised fix to effectively discard the excessive CFMs that are causing email delivery delays.

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.

Start time: Monday, July 17, 2023, at 3:00 PM UTC

Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.

Next update by: Tuesday, July 18, 2023, at 1:30 PM UTC

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.

Current status: We implemented an additional fix to prevent the Control Flow Messages (CFMs) that are impacting email delivery. We're continuing to monitor the environment to ensure that the delays do not reoccur. It is possible that during this time, users may see stale free/busy information.

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.

Start time: Monday, July 17, 2023, at 3:00 PM UTC

Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.

Next update by: Tuesday, July 18, 2023, at 8:30 PM UTC

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.

Current status: During our review of the system logs, we identified that Control Flow Messages (CFMs) are still sending at an increased rate. We're looking into options to prevent the additional CFMs in the affected portion of infrastructure, which we expect will mitigate the email delivery delays.

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.

Start time: Monday, July 17, 2023, at 3:00 PM UTC

Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.

Next update by: Tuesday, July 18, 2023, at 6:30 AM UTC

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.

Current status: We're continuing to monitor the environment to verify that messages are delivering successfully. We're still seeing some issues with messages queueing for delivery on a portion of infrastructure. We're reviewing system logs to see if there are any message caches that are trying to resend messages.

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.

Start time: Monday, July 17, 2023, at 3:00 PM UTC

Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.

Next update by: Tuesday, July 18, 2023, at 4:30 AM UTC

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.

Current status: We're continuing to monitor to ensure messages deliver as expected, and for the majority of users impact has been resolved. There's a small portion of infrastructure still experiencing a queuing issue which we're examining further to understand why it's not seeing relief where others are and determine steps to ensure all messages are successfully delivered.

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.

Start time: Monday, July 17, 2023, at 3:00 PM UTC

Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.

Next update by: Tuesday, July 18, 2023, at 2:30 AM UTC

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.

Current status: We've completed our deployment of the fix and the reversion process has completed. We're monitoring mail queues to ensure they deliver as expected, resolving impact.

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.

Start time: Monday, July 17, 2023, at 3:00 PM UTC

Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.

Next update by: Tuesday, July 18, 2023, at 12:30 AM UTC

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.

Current status: We've identified a recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure which is resulting in message queuing. We're working to disable this change so no further mail becomes queued and determine if additional steps are needed to drain the current backlogged messages.

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.

Start time: Monday, July 17, 2023, at 3:00 PM UTC

Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.

Next update by: Monday, July 17, 2023, at 10:30 PM UTC

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.

Current status: We're reviewing service health telemetry information to identify possible sources of impact where we see mail queueing occurring to help formulate our next troubleshooting steps.

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.

Next update by: Monday, July 17, 2023, at 8:30 PM UTC

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.

More info: The issue isn't client-specific, thus may occur in all Exchange Online connection methods.

Current status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.