Performance Issue - Map Rendering Delayed on US Cluster Customers

The impact of any remaining lag is negligible for customers on the US2 cluster and should resolve itself.

All other clusters are running optimally.

We are closing this incident at this time.

A Root Cause Analysis (RCA) will follow after completing a full review.

The US1 cluster has fully recovered.

Most of the US2 cluster’s clients have fully recovered. The delay is only in processing interface information in the map and only applies to a small subsection of clients. The estimate for recovery on this final part of the data lag for this map component depends on the influx of data it receives today.

All other parts of the product are running normally.

We continue actively monitoring the situation while waiting for this final component to recover from its data lag.

We understand the impact of this incident on your experience with the product and sincerely apologize for the inconvenience it has caused.

We’ve identified the source of the performance issue with Map discovery and rendering for customers running under the US1 and US2 clusters. We are waiting as the informational lag works through the data backlog.

The US1 cluster has had most of its clients fully recover. However, a very small subsection of clients still has a data lag and is delayed. Due to a heavy influx of data, the cluster is processing the data but maintaining the backlog size. The delay is only in processing interface information in the map.

The US2 cluster is still delayed but is still decreasing in lag. Customers are only experiencing interface information delays in the map.

We anticipate a full recovery by 11:00 UTC (7:00 EDT) tomorrow.

Rest assured, the dashboard information and alerts remain unaffected, providing up-to-date and accurate information.

We are diligently and actively monitoring the situation. We are waiting for the remaining components to catch up and be current.

We understand the impact of this incident on your experience with the product and sincerely apologize for the inconvenience it has caused.

We’ve identified the source of the performance issue with Map discovery and rendering for customers running under the US1 and US2 clusters. We are waiting as the informational lag works through the data backlog.

The US1 cluster has had most of its clients fully recover. However, a very small subsection of clients still has a data lag and is delayed. Due to a heavy influx of data, the cluster is processing the data but maintaining the backlog size. The delay is only in processing interface information in the map.

The US2 cluster is still delayed but is still decreasing in lag. Customers are only experiencing interface information delays in the map.

Rest assured, the dashboard information and alerts remain unaffected, providing up-to-date and accurate information.

We are diligently and actively monitoring the situation. We are waiting for the remaining components to catch up and be current.

We understand the impact of this incident on your experience with the product and sincerely apologize for the inconvenience it has caused.

We’ve identified the source of the performance issue with Map discovery and rendering for customers running under the US1 and US2 clusters. We are waiting as the informational lag works through the data backlog.

The US1 cluster has almost recovered, with only a small subset of customers experiencing interface information delays in the map.

The US2 cluster is still delayed, customers are only experiencing interface information delays in the map.

Customers on the US3 and US5 clusters have fully recovered since the last update.

Dashboard information and alerts are not affected and are providing up-to-date information.

We are actively monitoring the situation and waiting for the remaining components to catch up and be current.

We understand the impact of this incident on your experience with the product and we sincerely apologize for the inconvenience it has caused.

We’ve identified the source of the performance issue with Map discovery and rendering for customers running under the US clusters (US1, US2, US3, US5). We are waiting as the informational lag works through the data backlog.

Dashboard information and alerts are not affected and are providing up-to-date information.

The maps for customers on a small percentage of customers on the US5 cluster still show delayed inferred connections, but the rest of the map should be current. The inferred connection delay should conclude in the next several hours.

Clients on US clusters US1, US2, and US3 continue to decrease their lag. We now estimate it will take another 10-12 hours for all clusters' Map discovery and rendering to be current again. Several components are again current in the map. We are waiting for the remaining components to catch up and be current. We continue to monitor this.

We understand the impact this is having on your experience with the product and apologize for any impact this may be having on you and your clients.

We’ve identified the source of the performance issue with Map discovery and rendering for customers running under the US clusters (US1, US2, US3, US5). We are waiting as the informational lag works through the data backlog.

Dashboard information and alerts are not affected and are providing up-to-date information.

The maps for customers on the US5 cluster still show delayed inferred connections, but the rest of the map should be current. The inferred connection delay is still dropping and should become current in the next 4 hours.

Clients on US clusters US1, US2, and US3 are continue to decrease their lag. We now estimate it will take another 18-20 hours for all clusters' Map discovery and rendering to be current again. We will continue to monitor this.

We understand the impact this is having on your experience with the product and apologize for any impact this may be having on you and your clients.

We’ve identified the source of the performance issue with Map discovery and rendering for customers running under the US clusters (US1, US2, US3, US5). We are waiting as the informational lag works through the data backlog.

Dashboard information and alerts are not affected and are providing up-to-date information.

The maps for customers on the US5 cluster still show delayed inferred connections, but the rest of the map should be current. The inferred connection delay is still dropping and should become current in the next 4-8 hours.

Clients on US clusters. US1, US2, and US3 are continuing to decrease their lag. We estimate it will take another 24 hours for all clusters' Map discovery and rendering to be current again. We will continue to monitor it.

We apologize for the impact this may be causing you and your clients.

We’ve identified the source of the performance issue with Map discovery and rendering for customers running under the US clusters (US1, US2, US3, US5). We are waiting as the informational lag works through the data backlog.

Dashboard information and alerts are not affected and are providing up-to-date information.

We still expect clients on the US5 cluster to recover from their lag sometime during the evening, most likely in the next four hours.

Clients on US clusters. US1, US2, and US3 are slowly decreasing their lag. We do not have an estimate of when their Map discovery and rendering will be current, but we continue monitoring it closely.

We apologize for the impact this may be causing you and your clients.

We continue to monitor progress and will post relevant updates.

We’ve identified the source of the performance issue with Map discovery and rendering for customers running under the US clusters (US1, US2, US3, US5). We are waiting as the informational lag works through the data backlog.

Dashboard information and alerts are not affected and are providing up-to-date information.

We expect clients on the US5 cluster to recover from their lag at some point during the evening.

Clients on US clusters. US1, US2, and US3 are slowly decreasing their lag. We do not have an estimate of when their Map discovery and rendering will be current, but we continue monitoring it closely.

We apologize for the impact this may be causing you and your clients.

We continue to monitor progress and will post relevant updates.

We’ve identified the source of the performance issue with Map discovery and rendering for customers running under the US clusters (US1, US2, US3, US5). We are waiting as the informational lag works through the data backlog.

The lag for clients on the US4 cluster should be recovered in the next hour.

Dashboard information and alerts are not affected and are providing up-to-date information.

We apologize for the impact this may be causing you and your clients.

We continue to monitor progress and will post updates throughout the delay.

We’ve identified the source of the performance issue with Map discovery and rendering for customers running under the US clusters (US1, US2, US3, US4, US5). We are waiting as the informational lag works its way through the data backlog.

Dashboard information and alerts are not affected and are providing up-to-date information.

All relevant resources have been upgraded to provide the most expedient resolution.

We apologize for the impact this may be causing you and your clients.

We will continue to monitor the progress and post updates throughout the day.