Azure OpenAI Service – Global Latency Issues

We have been monitoring the latency issues that were affecting the AI Design Assistant yesterday. As previously mentioned, Microsoft made configurations changes yesterday, which resolved the issues. Thank you for your patience and understanding.

At 16:01 UTC, Microsoft provided a further update. Microsoft identified that a recent deployment introduced a misconfiguration that inadvertently led to an increase in traffic, uneven distribution and placed a limit on backend instances upon which Azure Open AI services rely. This manifested in users in East US experiencing latency and availability problems. Microsoft implemented configuration changes to remove these limits and manually rebooted the impacted instances to ensure the changes were applied.

At 16:35 UTC, Microsoft provided a further update that the issue is now mitigated. Here at Anthology, we are performing testing on Blackboard instances globally and can see that latency has improved.

We will continue to monitor the situation closely over the coming minutes and hours. We will keep this open for monitoring at this time, though we expect it to be resolved.

At 15:04 UTC, Microsoft provided an update. The rollback to a previous, known build has not alleviated latency in the US East region of the Azure Open AI Service. Microsoft are focussing efforts on identifying an appropriate mitigation path to resolve this issue.
We will continue to provide updates as more information becomes available and again, appreciate your patience.

At 10:14 UTC, we identified a latency issue affecting the AI Design Assistant, resulting in delayed or failed AI responses. This appears to be related to an ongoing issue with the Microsoft Azure OpenAI Service in the US East region.
Microsoft is actively investigating the problem and is rolling back a recent update that may be contributing to the service disruption. We will continue to provide updates as more information becomes available.
We apologize for any inconvenience this may cause and appreciate your patience...