Ingestion delays and failing queries in US

Ingestion has caught up and everything is back to normal.

The query load issue is resolved and queries should be loading as normal. We're still working through the backlog of events, so some charts might be showing data that is ~30 minutes old. That should be resolved soon.

No data was lost.

Cluster is looking stable now and we have been able to resume the ingestion of events.

Query performance should be now back to normal, as we are not hitting the limits anymore.

There is still events lag that we are already consuming, so data shown won't be up to date yet. We'll send an update once it's completely recovered.

No data has been lost during this period.

We are continuing to see queries failing and ingestion lag. We are switching on more capacity, which should hopefully resolve the queries failing (though they'll be showing data that is ~30-60 minutes out of date).

This is not impacting workflows or CDP, and no data has been lost.

We will keep you up to date.

We are under heavy load, and seeing query outages and delayed ingestion of events. No data is lost.

Our ClickHouse cluster is going under heavy load right now and the ingestion of events is being delayed.

We have found the root cause and are now working to leave the cluster in a stable state so we can catch up on the ingestion.