Delayed cluster modifications

This incident has been resolved.

Recovery procedures are complete, and we are continuing to monitor closely.

We continue to recover. Cluster operations submitted between 18:45 UTC and 21:55 UTC may continue to be delayed or delay subsequent changes. All other cluster operations submitted from this point forward should execute normally. Normal cluster operation is expected to resume within the next hour (by 00:00 UTC). Lower granularities of metrics for recent time periods may continue to be unavailable for the next five hours (until 04:00 UTC).

We are continuing to execute recovery procedures. Data Explorer and Real Time Performance Panel are fully operational. Cluster operations are recovering but still delayed. Lower granularities of metrics for recent time periods may continue to be unavailable. We will post another update in 30 minutes.

We are continuing to investigate this issue.

We have identified the root cause and have begun the recovery procedure. We will post another update within the next 30 minutes.

We are continuing to investigate and will post another update in 30 minutes.

We are continuing to investigate and will post another update in 30 minutes

We have identified that usages of Data Explorer may be delayed or unresponsive during this time. Additionally, certain operations in the Real-Time Performance Panel may not work. Lower granularities of metrics for recent time periods may also be unavailable.

We are currently investigating an issue that is causing delays in all cluster updates across all cloud providers.