PostHog US Outage History

PostHog experienced an issue causing analytics queries to timeout or fail, affecting dashboards, insights, and other query-driven views which became slow or unavailable. The incident lasted 1.1 hours and was classified as minor severity. The service was restored after the query performance issues were resolved.

PostHog's web application was inaccessible for 55 minutes due to access issues. The service has been restored and the team is monitoring performance to ensure stability.

PostHog experienced a 14.8-hour incident where a sharp spike in query volume caused database overload, resulting in failed queries, dashboard loading errors, and event ingestion lag for customers in the US region. The team restarted affected services and adjusted application retry behavior to restore functionality, while continuing to monitor and investigate the underlying cause of the query volume spike.

PostHog's web app experienced accessibility issues that prevented users from accessing the service. The incident was classified as minor and lasted 45 minutes. The team investigated and resolved the access problems, restoring normal service functionality.

PostHog's feature flags API experienced elevated error rates for 55 minutes, causing flag evaluations to fail or fall back to default behavior for some requests. The issue was caused by increased database load, which was resolved by provisioning a new database instance. API request responses returned to normal after the new database resources were deployed.

PostHog AI experienced elevated API error rates caused by upstream provider issues. The service disruption lasted 1.6 hours while the team monitored the situation. This was classified as a minor incident affecting the AI functionality within PostHog.

PostHog experienced processing delays in their event ingestion pipeline at the final Clickhouse ingestion stage, causing events to take longer than usual to appear in the product. While no data was lost, users experienced delays in seeing events show up in PostHog apps and queries. The processing delays were resolved after 1.4 hours with processing times returning to normal levels.

PostHog experienced a 15.1-hour event ingestion processing delay caused by a degraded shard during routine maintenance and elevated part counts on ClickHouse, which led to insert rejections and Kafka consumer lag. Events took longer than usual to appear in PostHog apps and queries, though no data was lost during the incident. The root cause was identified and resolved, with ingestion resumed and the backlog processed over approximately 2 hours.

PostHog's logs ingestion service in the US region experienced a database corruption issue that initially caused 30-minute delays but escalated to complete data loss for all customer logs prior to February 16th, 21:00 UTC. The incident affected only the new Logs product, while other PostHog services like analytics events and replays remained unaffected due to isolated infrastructure. PostHog restored the service by removing corrupted database files and successfully backfilled available backup data within 24 hours, returning the logs service to normal operation.