On-call notification delays

We have now sent one notification to each escalation path affected by the incident this morning, with a link to review any expired escalations in more detail. We’ve chosen to roll these up, to avoid users receiving a large amount of stale notifications at once.

For now, we'll continue to monitor very closely, whilst the AWS incident is ongoing.

If you require further support, please reach out to us on Slack, or via support@incident.io "support@incident.io": http://support@incident.io/

We have reviewed all delayed notifications from the incident this morning, and will shortly be sending one notification to each affected escalation path, with a link to review any expired escalations in more detail

We’ve chosen to roll these up, to avoid users receiving a large amount of stale notifications at once.

We'll confirm once this has been done. If you require further support, please reach out to us on Slack, or via support@incident.io: mailto:support@incident.io

Escalations are continuing to process as normal, in real-time.

We're continuing to work on processing delayed notifications, with further updates to follow.

Escalations are now sending as normal, in realtime and we are seeing platform recovery across the board.

For previously delayed notifications, we are continuing to work on ensuring they are all delivered as intended, though there may be a further delay of up to 30m.

Once we've confirmed full recovery, we'll send a further update.

We're still seeing severe delays with sending escalations. We're monitoring the upstream AWS outage closely and exploring multiple mitigation options to work around affected upstream providers. Further updates to follow.

We've identified the source of the delayed escalation notifications. Several upstream vendors are currently being affected by a currently ongoing and widespread AWS outage. Our team are continuing to mitigate and restore normal notification service levels.

More details here: https://health.aws.amazon.com/health/status

We'll provide updates every 30m going forward.

We are investigating an issue with delays on escalation notifications - our engineers are investigating and will update shortly.