Workflows: Email sending currently delayed

Email sending is currently delayed. We are investigating the issue. No data is lost.

We found the culprit which turns out to be a poison pill entry in our email queue which blocks the queue from proceeding. We are taking steps to resolve this issue.

We fixed the underlying issue and email sending is proceeding normally again. Emails will be delayed. We are monitoring the fix and taking action to avoid a poison pill like this to happen again.

Whilst the temporary fix worked we have meanwhile taken action and hardened our systems so this kind of poison pill problem can't happen again. The email queue will drain and some emails will arrive later but no data was lost.

We're still working through the backlog of emails - we're monitoring and will update this incident once we've caught up. You can monitor progress of individual workflows from the 'invocations' tab of the workflow in PostHog - delayed runs will transition from 'Running' to 'Succeeded' once complete.