Hierarchical data based on user roles (sub-sites, user roles, etc.) are impacting Services

The incident has been fully resolved. Regular service has been restored, and all systems are operating as expected.

Impact:
Users should no longer experience any issues related to this incident.

Service has been fully restored. We apologize for the degradation in services. We thank you for your understanding. If you continue to experience issues, please don't hesitate to contact our support team.
We will post an RCA after an internal investigation.

Our team has implemented fixes for the ongoing issues. We are still in recovery mode.

Impact:
We continue to address the ancillary problems from this incident. We are continuing to address issues to ensure stability.

Next Steps:
We request that if you are still experiencing issues with your tenant(s), contact support.

We appreciate your continued patience as we work through this issue.

Our team has implemented fixes for the ongoing issues. We are still in recovery mode.

Impact:
We continue to address the ancillary problems from this incident. We are continuing to address issues to ensure stability.

Next Steps:
We request that if you are still experiencing issues with your tenant(s), contact support.

We appreciate your continued patience as we work through this issue.

Our team has implemented fixes for the ongoing issues. We are still in recovery mode.

Impact:
The issues regarding site hierarchy and user access to sites, along with sites disassociated from their parent site, have been resolved. We continue to address the ancillary problems from this incident.

Next Steps:
We request that if you are still experiencing issues with your tenant(s), contact support.

We appreciate your continued patience as we work through this issue.

Our team has implemented fixes for the ongoing issues. We are in recovery mode, but we are still addressing a few outstanding problems.

Impact:
There is still an outstanding issue regarding site hierarchy and user access to sites. This is a rendering issue. The sites are not disassociated from their parent site in the back-end tables.
The All Sites pages are not loading data for some clients.
Significant progress has been made overnight in fixing the sites affected.
Alerts and monitoring are working for both of these ongoing issues.

Next Steps:
We will continue to work on these issues and will provide updates as they are relevant.
We expect continued progress throughout the morning.

We appreciate your continued patience as we work through this issue.

Our team has implemented fixes for the ongoing issues. We are in recovery mode, but we are still addressing a few outstanding problems.

Impact:
There is still an outstanding issue regarding site hierarchy and user access to sites. This is a rendering issue. The sites are not disassociated from their parent site in the back-end tables.
The All Sites pages are not loading data for some clients.
Alerts and monitoring are working for both of these ongoing issues.

Next Steps:
We will continue to work on these issues and will provide updates as they are relevant.
We expect continued progress throughout the morning.

We appreciate your patience as we work through this issue.

Our team has implemented fixes for the ongoing issues. We are in recovery mode, but we are still addressing a few outstanding problems.

Impact:
There is still an outstanding issue regarding site hierarchy and user access to sites. This is a rendering issue. The sites are not disassociated from their parent site in the back-end tables.
The All Sites pages are not loading data for some clients.
Alerts and monitoring are working for both of these ongoing issues.

Next Steps:
We will continue to work on these issues and will provide updates as they are relevant.
We expect continued progress throughout the evening.

We appreciate your patience as we work through this issue.

Our team has implemented fixes for the ongoing issues. We are in recovery mode, but we are still addressing a few outstanding problems.

Impact:
There is still an outstanding issue regarding site hierarchy and user access to sites. This is a rendering issue. The sites are not disassociated from their parent site in the back-end tables.
The All Sites pages are not loading data for some clients.
Alerts and monitoring are working for both of these ongoing issues.

Next Steps:
We will continue to work on these issues and will provide updates as they are relevant.
We expect continued progress throughout the day today.

We appreciate your patience as we work through this issue.

Our team has implemented fixes for the ongoing issues. We are in recovery mode, but we are still addressing a few outstanding problems.

Impact:
There is still an outstanding issue regarding site hierarchy and user access to sites. This is a rendering issue. The sites are not disassociated from their parent site in the back-end tables.
The All Sites pages are not loading data for some clients.
Alerts and monitoring are working for both of these ongoing issues.

Next Steps:
We will continue to work on these issues and will provide updates as they are relevant.
We expect continued progress throughout the day today

We appreciate your patience as we work through this issue.

Our team has implemented fixes for the ongoing issues. We are in recovery mode, but we are still addressing a few outstanding problems.

Impact:
Users now have access to their client in the US5 cluster.
Shared collectors now have connectivity.

There is still an outstanding issue regarding site hierarchy and user access to sites.
Alerting and monitoring are working for those sites.

Next Steps:
We will continue to work on these issues and will provide updates as they are relevant.

We appreciate your patience as we work through this issue.

Affected Services: Services based on hierarchical data, based on user roles (sub-sites, user roles, shared collectors)
Cluster(s): Several Cluster

Our team has identified the root cause of the degraded performance and is currently investigating a solution to restore normal service levels.

Impact:
While we work on the resolution, users may continue to experience performance issues.
Issues with email alerts, shared clusters, and login issues on US5 are being addressed.
Data display or data displaying incorrectly is still under investigation

Next Steps:
We will provide updates as the situation progresses.

Your patience is greatly appreciated, and we regret any inconvenience you may be experiencing.

Affected Services: Services based on hierarchical data, based on user roles (sub-sites, user roles, shared collectors)
Cluster(s): Several Cluster

Our team has identified the root cause of the degraded performance and is currently investigating a solution to restore normal service levels.

Impact:
While we work on the resolution, users may continue to experience performance issues.
Correction to the prior post, sites with disconnected shared collector issues are not currently being monitored.

Next Steps:
We will provide updates as the situation progresses.

Your patience is greatly appreciated, and we regret any inconvenience you may be experiencing.

Affected Services: Services based on hierarchical data based on user roles (sub-sites, user roles, etc.)
Cluster(s): Appears to include several sites

We are investigating reports of degraded performance. Our team is working diligently to identify the root cause and restore optimal performance.

Impact:
While we address the situation, users may experience access to some functionality with sites and sub-sites

Services: Monitoring and alerting are currently working as expected

Next Steps:
We will provide updates as we learn more.

We appreciate your patience as we work to resolve this issue.

Affected Services: Services based on hierarchical data based on user roles (sub-sites, user roles, etc.)
Cluster(s): Still determining. More than just US4

We are investigating reports of degraded performance. Our team is working diligently to identify the root cause and restore optimal performance.

Impact:
While we address the situation, users may experience access to some functionality with sites and sub-sites

Services: Monitoring and alerting are currently working as expected

Next Steps:
We will provide updates as we learn more.

We appreciate your patience as we work to resolve this issue.

We are continuing to verify sites and restore access. Some users may still experience issues with data not loading on their sites. Monitoring is not impacted.

Most sites are now operational. However, some users are still experiencing issues with site loading or logging in. Our team is actively working to resolve these remaining problems. Monitoring is not impacted. We appreciate your patience.

Auvik will undergo emergency maintenance. During this time, you may not be able to log into Auvik. There may also be interruptions to your network monitoring.