Analytics: query timeouts/failures

We're continuing to monitor the occasional spikes in query timeouts and errors. Recovery from the spikes has been happening more quickly.

We're still seeing occasional spikes in query timeouts and errors, but recovering much more quickly. We're continuing to monitor and investigate.

We've reopened this incident after seeing the spikes in query timeouts/failures return. Apologies for the ongoing troubles. We're are investigating again.

We've resolved the query timeouts/failures incident. Cluster stability has improved and recent configuration changes to table profiles have been applied. Queries and the app are operating normally, with no backlog observed in distribution queues after the updates.

We've seen some smaller spikes in failed queries, so we're still monitoring this closely for a while, just in case, before we call this one resolved.

Load has been looking much better, but we are still monitoring for a while before we mark this one as resolved. Thanks much for your patience.

We are still monitoring the cluster to ensure stability. The failure rate for insight queries has dropped, and customer-facing query errors have decreased, but we are continuing to watch for further issues.

We have resumed event ingestion and are monitoring load on the cluster

We fixed the root cause and are monitoring the load on our ClickHouse cluster.

We have identified the issue and we are working on the fix

Unfortunately, the issue has reoccured and we have begun a second investigation.

-

We’ve identified an issue affecting analytics queries (timeouts and/or failures). Dashboards, insights, and other query-driven views may be slow or unavailable.

Ingestion is also impacted, so data delays are expected when issuing queries.

The load spike has been resolved and systems are operating normally.

We’ve identified an issue affecting analytics queries (timeouts and/or failures). Dashboards, insights, and other query-driven views may be slow or unavailable.

Ingestion is also impacted, so data delays are expected when issuing queries.