Blackboard Email Notification Failures

This incident has been resolved.

We are continuing to monitor all activity with emails. Email sending and email notifications are working as expected at this time.

We are continuing to monitor all activity with emails. Email sending and email notifications should be working as expected at this time.

We are proactively monitoring email issues and noticed that we experienced an outage for about 30 minutes at around 9:30 am EST. Emails should now be back up and running.

While we continue to work towards final resolution of the hardware failure impacting emails, we may experience some periods of email delays or outage. We are monitoring all activity, during this period and for the foreseeable future and if we notice outages we will quickly work to mitigate the risks.

We continue to monitor the email sending infrastructure as we work towards final resolution of the hardware failure impacting email notifications. At this time, we are not seeing any email notification failures.

As part of today's (October 10th) maintenance update, a fix is also being released that will prevent the retried notifications from being received by administrators.

We continue to monitor the email sending infrastructure as we work towards final resolution of the hardware failure impacting email notifications. At this time, there should be minimal email notification failures.

For administrators receiving user’s email notifications, these notifications should be decreasing as email notifications recover. We are planning to release a fix that will prevent these retried notifications from being received by the administrator as part of the next maintenance update on Thursday, October 10th.

We have completed mitigations for the email notification failures. We are continuing to monitor as we work towards final resolution.

For the user email notifications being received by administrators, we expect to see fewer of these as email notifications recover. We plan to roll out further fixes as part of the maintenance release on Thursday, October 10th.

For the email notification issue where administrators were incorrectly BCC’d on user emails, this issue has been fixed and is scheduled to be deployed to Production by this Thursday, October 10th.

For the sporadic email send failures, we have identified the issue. We are in the process of mitigating the issue. A further update will be provided as more information becomes available and as we make progress.

Thank you for your continued patience.

For the email notification issue where administrators were incorrectly Bcc’d on user email, this issue has been fixed and is scheduled to be deployed to Production by this Thursday. Also, the resolution of the ongoing infrastructure issue remains in progress, a further update will be provided as more information becomes available.

Our teams are continuing to investigate and resolve infrastructure issues resulting in sporadic email notification failures. In addition, we are working on a fix for administrators receiving user's notification emails, and expect this fix to be included in an upcoming maintenance release.

Thank you for your continued patience as we address these issues.

For the unexpected behavior sending certain notification emails, multiple teams are continuing to investigate the issue. For the issue with administrators receiving user's notification emails, our Development team is working on code changes.

Further investigations have identified the email notification issue was due to the infrastructure becoming unresponsive. Email infrastructure has been restarted to resolve the issue.

Multiple teams continue to investigate earlier reports of email notification failures. A further update will be provided as more information becomes available.

We are aware of some ongoing email notification failures. We are investigating and will provide updates as more information is available.