Cluster CA1 Sites Unavailable - New Site creation access across all clusters impacted along with Okta account access

Our team has successfully completed the out-of-band maintenance on the EU1 cluster. We will continue to monitor EU1 closely throughout the evening to ensure stability.

The earlier slowness affecting clusters outside of CA1 now appears to be resolved. We will continue to validate system performance and client connections across all clusters.

Impact:
EU1: Sites remain stable following maintenance; we are monitoring for any recurrence.
Other Clusters: No ongoing slowness observed.
Login / Okta: A small subset of clients using Okta may still experience denied login attempts; a password reset may be required.

Next Steps:
Continue overnight monitoring of EU1 and client connection stability.
Perform additional validation tomorrow on site performance and login flows.

If you encounter related issues, please contact Auvik Support so we can track and assist further.

Next Steps:
A final update will be posted once we confirm the resolution.

Our team has identified a suspected cause of the slowness on the EU1 cluster and is taking steps to remediate the issue by performing an out-of-band maintenance on the EU1 cluster.

Impact:
Customers may experience issues accessing their sites on the EU1 cluster over the next two hours while this maintenance is in progress.

Sites on the other clusters are not affected by the action.

Please report any related issues to Auvik Support so we can track and assist further.

Next Steps:
We are applying mitigation measures and will provide updates on progress.

We are continuing to monitor for any further issues.

We continue to experience system slowness on clusters outside of CA1, which is attributed to residual cleanup efforts related to earlier issues in the CA1 cluster. Although our team has made progress, some symptoms persist, and we continue to work on resolving them.

Impact:
Performance: Some clients may still see slower performance.
Site Access: Accessing sites via xxx.my.auvik.com may fail. Sites remain accessible using the fully qualified URL format (xxx.[cluster].my.auvik.com).
Login Issues: Some sites are still experiencing login issues.
Okta Authentication: A small subset of clients using Okta may continue to see denied login attempts. A password reset may be required.

Next Steps
We are focused on restoring full login functionality and addressing the residual impacts of the cleanup. A final update will be provided once all issues are fully resolved.

If you continue to experience problems, please contact Auvik Support.

We are currently experiencing some system slowness on clusters outside of CA1. This is related to residual cleanup from earlier issues on the CA1 cluster. Our team is actively addressing this to ensure ongoing platform stability.

Additionally, the cleanup contributed to a site redirection issue, where sites may not be accessible via xxx.my.auvik.com. These sites are accessible when using the fully qualified URL (e.g., xxx.us1.my.auvik.com), where us1 reflects the cluster hosting the site. In some cases, newly created sites may not display correctly.

A small percentage of clients using Okta authentication were also impacted. If you encounter a denied login message, you may need to reset your password by following the provided instructions.

Impact
Some clients may see slower performance.
Site access via xxx.my.auvik.com may fail; use the direct cluster URL (xxx.[cluster].my.auvik.com) as a workaround.
Limited Okta users may experience authentication issues requiring a password reset.

Next Steps
Our engineering team is continuing remediation efforts. A final update will be provided once the issues are fully resolved.

If you continue to encounter problems, don't hesitate to get in touch with Auvik Support
.

We have applied changes to address the issue with new site creation. We continue to experience issues with sites not being visible promptly. We are continuing to work on this issue.

This is also contributing to a site redirection issue where sites are not accessible via xxx.my.auvik.com, but are accessible via xxx.us1.my.auvik.com, where us1 is the cluster on which the site is located.

Other services appear to be operating normally, and sites are becoming visible in the UI. We are monitoring closely for stability.

Impact:
We are continuing to experience issues with a small percentage of Okta-enabled accounts that lack access. We are remediating these accounts.
If you continue to encounter problems, please report them to Auvik Support.

Next Steps:
A final update will be posted once we confirm the resolution.

We have applied changes to address the issue with new site creation. We continue to experience issues with some cases of child site creations and are working to rectify them.

Other services appear to be operating normally, and sites are becoming visible in the UI. We are monitoring closely for stability.

Impact:
All other services should be operating normally for the tenants in the CA1 cluster.
We are continuing to experience issues with a small percentage of Okta-enabled accounts that lack access. We are remediating these.
If you continue to encounter problems, please report them to Auvik Support.

Next Steps:
A final update will be posted once we confirm the resolution.

We are currently investigating reports of not being able to access new sites after they are created. This is under all clusters.

Impact:
All other services should be operating normally for the tenants in the CA1 cluster.
We are continuing to experience issues with a small percentage of Okta-enabled accounts that lack access. We are looking to remediate these.
If you continue to encounter problems, please report them to Auvik Support.

Next Steps:
We will be monitoring the situation during the evening and will report updates in the morning.
A final update will be posted once we confirm the resolution.

We have implemented changes to address the issue with services for clients on the CA1 clusters, and these clusters are now operating normally. We are monitoring closely for stability.

Impact:
Services should be operating normally for the tenants in the CA1 cluster.
Tenants created have also been restored.
There are still issues with some clients who use Okta to access Auvik. We are looking to remediate these.
If you continue to encounter problems, please report them to Auvik Support.

Next Steps:
We will be monitoring the situation during the evening and will report updates in the morning.
A final update will be posted once we confirm the resolution.

Our team has identified a suspected cause of the sites on the CA1 cluster being inaccessible and is taking steps to remediate the issue.
Sites have been restored to yesterday. We are working to restore data that was present at the time of this incident.

There have also been reports of some users experiencing issues accessing Auvik using their Okta setup. We are working to restore access.

Additionally, any new tenants or users created since 17:15 UTC (1:15 PM ET) are currently not functioning or accessible. Tenant restoration is occurring.

Impact:
Customers should now be able to access their sites on the CA1 cluster, with the exceptions noted above. Data restoration is being worked on.
Please report any related issues to Auvik Support so we can track and assist further.

Next Steps:
We are applying mitigation measures and will provide updates on progress.

Our team has identified a suspected cause of the sites on the CA1 cluster being inaccessible and is taking steps to remediate the issue.
There have also been reports of some users experiencing issues accessing Auvik using their Okta setup. This is also being investigated.
Additionally, any new tenants or users created since 17:15 UTC (1:15 PM ET) are currently not functioning or accessible.

Impact:
Customers may continue to experience inaccessibility to their site on the CA1 cluster as the cluster comes back up.
Clients or clusters created since the incident began are not accessible.
Please report any related issues to Auvik Support so we can track and assist further.

Next Steps:
We are applying mitigation measures and will provide updates on progress.

Our team has identified a suspected cause of the sites on the CA1 cluster being inaccessible and is taking steps to remediate the issue.
There have also been reports of some users experiencing issues accessing Auvik using their Okta setup. This is also being investigated.

Impact:
Customers may continue to experience their site being inaccessible on the CA1 cluster.
Please report any related issues to Auvik Support so we can track and assist further.

Next Steps:
We are applying mitigation measures and will provide updates on progress.

We are continuing to investigate tenants on the CA1 cluster that are not accessible.

Impact:
Customers are experiencing an interruption of access to their sites.
The following services are not affected: Other clusters are not affected by this outage.

Next Steps:
Our team is working to identify contributing factors. Updates will follow as more information becomes available.

We are continuing to investigate this issue.

We are currently investigating reports of a clients not being able to access their tenants on the CA1 cluster.

Impact:
Customers may experience an interruption of access to their sites.
The following services are not affected: Other clusters are not affected by this

Next Steps:
Our team is working to identify contributing factors. Updates will follow as more information becomes available.