High failure rates calling Authorization service in East US Delayed Update region

This incident has been resolved.

The system has been stable after applying the mitigation. The investigation is still in progress to determine the root cause. We will provide further updates as we gain more insights.

Thank you for your patience.

Two instances of our backend authorization service had their connection pools run out of available connections. We restarted the two service instances as a short term mitigation, and increased the connection pool size limit as a longer term mitigation.

A high percentage of requests to our backend authorization service in the delayed update ring are failing. This is causing issues with filtering tenants in the portal, other services being able to authorization requests, and with enumerating entities in solutions builder flows. Currently it appears this is being caused by a failure to communicate with our backend database in the region.