Query performance degraded in US

One cell of our CH infrastructure encountered replication issues and required re-replication of the data in a shard. That is now complete and all systems are operating as expected. Enjoy your weekend and holidays!

Still monitoring for bit longer before we raise the API query limits again, but recovery is looking good.

We've applied some fixes, and are almost fully recovered. We're monitoring closely for a bit longer to be sure.

We're still working on this. You may see errors about busy query endpoints in your insights and reports in the app. This only impacts loading existing data from PostHog, ingestion and feature flags are unaffected.

To mitigate the impact from this incident, we have decided to limit API queries temporarily. You may experience increased rate limits on all public endpoints that fetch data from PostHog until we resolve this.

We have spotted a load spike in our ClickHouse cluster, we already identified the problem and are working to resolve it as soon as possible.

Query performance may be degraded and the app may not respond as expected.