NSZ PRD – Tenants are not accessible.

Service have been failed back over to PRD. All services are healthy and available. If any issues are noticed please engage customer support.

We have successfully failed over to the DR environment, and tenant services are continuing to come back online. We are actively monitoring system stability and performance.

Our DBA team is currently working on the failback process to the primary production environment. This process is taking longer than initially expected. Currently we are looking at completing the needed work and being ready to move services back to the primary datacenter overnight during the early morning hours of March 19th AEDT time.

Please note the following are currently impaired:
Reporting services will remain unavailable until production database services are fully restored.
Response times are likely to be slower in automation and web UI

We will continue to provide updates as more information becomes available.

Note: The failback process to the original environment will be seamless, and no action is required from customers during or after this process.

Once we are confident in the timings of rolling back we will update both Status Page and Cloud Calendar with the specific date/time.

We have successfully failed over to the DR environment, and tenant services are continuing to come back online. We are actively monitoring system stability and performance. Some performance degradation may be experienced as services stabilize in the DR environment.

Our DBA team is currently working on the failback process to the primary production environment. This process is taking longer than initially expected, and at this time we do not have an estimated timeline for completion.
Reporting services will remain unavailable until production database services are fully restored. We will continue to provide updates as more information becomes available.

Note: The failback process to the original environment will be seamless, and no action is required from customers during or after this process.

We have successfully failed over to the DR environment and tenant services are continuing to come back online. We are actively monitoring system stability and performance. Some performance degradation may be experienced as services stabilize in the DR environment.

Our DBA team is currently working on failover environment fixes. This failover process is taking longer than initially expected, and at this time we do not have an estimated timeline for completion.

Reporting services will remain unavailable until production database services are fully restored. We will continue to provide updates as more information becomes available.

We are continuing to monitor for any further issues.

We have successfully failed over to the DR environment and tenant services are continuing to come back online. We are actively monitoring system stability and performance. Some performance degradation may be experienced as services stabilize in the DR environment.
Our DBA team is currently working on failover environment fixes. This failover process is taking longer than initially expected, and at this time we do not have an estimated timeline for completion.
Reporting services will remain unavailable until production database services are fully restored. We will continue to provide updates as more information becomes available.

We encountered unexpected errors in the restoration of the data from the corrupted disks. As a result we will be pivoting to our Disaster Recovery site. This will result in roughly 15 minutes of data loss (data loss would be between 7:47 AM and 8:02 AM AEDT Mar 17, the 15 minutes leading up to the outage). Once we have completed the rerouting to the DR site we will update here. As a result of pivoting to the DR site reporting services will remain unavailable.

Working with Microsoft we have identified a fix and are in the process of implementing. Expected recovery is approximately 1 hour.

Microsoft Senior Engineers have engaged and are investigating. The work on reverting to the DR site is temporarily on hold while the Senior Engineer investigates as we have determined that there will be 15 minutes of lost data as part of the move to our DR site.

We are preparing to enable our DR location. The work to reroute traffic, enable data services, and restore tenant access is expected to take 1-1.5 hours to complete. There may be some period of degraded performance immediately following the rerouting of traffic. While the DR work is taking place the teams are continuing to work with our vendor in hopes they can resolve the issue faster without the need to invoke DR.

We have escalated and are working with our Service Provider to restore service.

Ivanti Engineering teams are engaged and working towards a resolution.

The issue has been identified and a fix is being implemented.

Tenants are temporarily not accessible. We are actively investigating this and will provide an update once we have more information.