PostHog US experienced a 6.4-hour incident affecting realtime destination delivery, caused by a poison pill message that clogged a single queue partition. Only that partition was impacted with event delays; no data was lost and the rest of the system remained healthy. A fix was developed and deployed, though its release was briefly delayed by a concurrent GitHub outage; the partition has since been confirmed as draining normally.
Trusted by 1,000+ teams
Stop finding out about outages from your users. Monitor 6,320+ cloud services and get alerted the second something breaks.
We’re investigating issues with realtime destinations. Some events may be delayed.
The problem was a poison pill message clogging the queue on one partition. Invocations on that particular partition see some delay whilst the rest of the system is completely healthy. No data was lost and delays on that partition is the only impact. We prepared a PR and are merging the fix.
We have a fix prepared but due to an github outage https://www.githubstatus.com/ we are waiting for actions to run to merge and resolve the issue.
The fix is deployed and the partition is now draining.
With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.
Start free trialNo credit card required · Cancel anytime · 6320 services available
Integrations with