Notification delay

This incident has been resolved.

The notification service has been restored, and notifications are now properly getting sent within the SLA. Scheduled notification jobs have been restored, so moving forward they will be sent at the scheduled time.

Scheduled notifications that were scheduled to be sent during the suspended period -- between 10:15a PDT to 11:23a PDT -- are being sent now.

We have partial recovery where new notifications are being sent within the SLA of 10 seconds. Scheduled notifications continue to be suspended.

We are temporarily suspending scheduled notification jobs in order to bring down the database load so that all other notifications -- including time-sensitive authentication notifications -- can recover to be delivered within SLA.

Customers may be experiencing a delay in notifications causing authentication delays. We are observing a 2 - 15 minute delay in notifications. This is under investigation.