PostHog EU's insight alert scheduler became overloaded by an unbounded batch of due alerts, causing scheduler timeouts and preventing new alert checks from being triggered. The issue affected the App component for approximately 10.6 hours, with alerts failing to send during this period. A fix was deployed to gradually drain the backlog, successfully resolving the issue for nearly all customers.
Trusted by 1,000+ teams
Stop finding out about outages from your users. Monitor 6,320+ cloud services and get alerted the second something breaks.
Alerts on insights are not being sent, the team is investigating
We’ve identified that the EU insight alert scheduler is getting overloaded by an unbounded batch of due alerts, which can cause scheduler runs to time out and prevent new alert checks from being scheduled. We have a mitigation ready in PR #97378 . This is intended to let the EU cluster drain the backlog gradually once deployed.
The fix to unblock the backlog is now deployed and in-progress. We're continuing to monitor the issue.
All but a handful of alerts are caught up. We're working to get those alerts processed now and continuing to monitor. For almost all customers there should be no impact at this time.
With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.
Start free trialNo credit card required · Cancel anytime · 6320 services available
Integrations with