Authentication Issues - System Impacting

Microsoft has indicated as of 00:05 UTC (6:05 MT) the impacted services have been mitigated.

Thank you.

-Cova Support

The latest update from Microsoft now indicates services should be fully restored by 00:40 UTC (6:40 MT). We are continuing to monitor impacted services and will provide an update once Microsoft indicates a full recovery.

The latest update from Microsoft now indicates services should be fully restored by 23:40 UTC (5:40 MT). We are continuing to monitor impacted services and will provide an update once Microsoft indicates a full recovery.

Microsoft has shared that their recovery effort involves reloading configurations and rebalancing traffic across a large number of nodes to restore full operational scale. The process is gradual by design, ensuring stability and preventing overload as dependent services recover. They are seeing strong signs of improvement across affected regions and are tracking toward full mitigation by 4:20 PST, today.

Customers should be seeing the initial signs of recovery with many already having services fully restored. We are continuing to monitor for updates from Microsoft and will provide another update when the issue is fully resolved.

No new updates from Microsoft have been provided. Some customers may be seeing the initial signs of recovery. We are continuing to monitor for updates from Microsoft and will share more information as it becomes available.

Continuing to use https://hub.iqmetrix.net/ as an entry point to HUB remains the best path to access back office functionality.

Microsoft has provided another update that indicates they are expecting services to fully recover over the next four hours as they continue to recover nodes. Some customers may begin to see initial signs of recovery. We are continuing to monitor for updates from Microsoft and will share more information as it becomes available and we start to see systems recover in the next few hours.

Continuing to use https://hub.iqmetrix.net/ as an entry point to HUB remains the best path to access back office functionality. For access to the Cova App - turning off the WiFi and using Offline Mode continues to be the best solution. See support article here for Offline Mode: https://support.covasoftware.com/covasupport/s/article/Offline-Mode-in-Cova

The latest update from Microsoft indicates they have pushed their "last known good" configuration and customers may begin to see initial signs of recovery. Traffic is being proactively routed through healthy nodes as others are in a recovery state. Continuing to user https://hub.iqmetrix.net/ as an entry point to HUB remains the best path to access back office functionality.

Our teams have deployed a fix that will grant access to the older version of HUB using this URL: https://hub.iqmetrix.net/

Access to reporting and other services hosted in New HUB will remain unavailable and users may experience performance issues.

We will continue to monitor Microsoft for updates and work internally to restore full connectivity.

The latest update from Microsoft does not indicate a timeline for restoration of services. Our teams continue to work on a deployment to reroute our authentication flows. Once deployed and tested we will advise of the change.

Microsoft has posted an update noting they are working on rolling back their AFD configuration to the last known good state. There is no timeline provided for the rollback but will provide additional updates within 30 minutes. Our internal teams are continuing to work on a solution to reroute our authentication services in the event Microsoft continues to experience issues.

Microsoft has provided an update noting they are actively accessing failover options of their internal AFD services and will provide additional information within 60 minutes. Our internal teams are exploring solutions to migrate our authentication hosting services to another platform while Microsoft explores additional recovery workstreams. We will continue to provide updates as we have more information.

Microsoft has noted Azure services began experiencing DNS issues at approximately 16:00 UTC resulting in a degradation of services. They have taken action that is expected to address the issues shortly. Updates from Microsoft are expected within the next 60 minutes.

Upon further investigation, we have identified a larger issue with our Microsoft Azure hosting services. We are monitoring their status and will continue to provide updates as they are provided.

We are currently experiencing a service disruption with authentication services impacting the ability to log into HUB. Users can continue to log into the tablet by turning off WIFI and using the bubble picker. We will continue to post updates as we work towards resolution.