We are currently investigating a problem regarding the performance of SMS and Voice notification delivery.

We are no longer experiencing any issues involving delayed or failed SMS or Phone call notification delivery.
Please contact Splunk On-Call support if you encounter any further problems.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

We are continuing to monitor for any further issues.

Call/Voice queues have now cleared and we are seeing voice notifications being delivered in real time again. Currently, all notification types are operational and working as expected. We are continuing to monitor notification rates and will provide further updates on this Status Page Incident as the ongoing AWS incident progresses.

Thank you for your continued patience while we work to resolve this issue.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

We are continuing to montior the situation. Currently SMS notifications are delivering as normal. However, Voice/Call notification are deeply backlogged and you may experience repeated calls for incidents that have already been actioned in your On-Call instance.

We are working directly with our notification provider to help migitate this isse as soon as possible.

Where possible, to avoid single points of failure, please take action to diversify your On-Call personal paging policies to include all forms of notification: SMS, Voice, Push, WhatsApp

Thank you for your contnued patience while we work to resolve this issue.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

AWS fix has been implemented.

We are seeing the successful delivery of SMS and Voice through our notification provider. However, there still may be a backlog of notifications to be processed.

Moving incident status to Monitoring.

Where possible, to avoid single points of failure, please take action to diversify your On-Call personal paging policies to include all forms of notification: SMS, Voice, Push, WhatsApp

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Our cloud telephony partner (SMS and Voice) is seeing partial recovery and is scaling their systems to handle the influx of traffic and processing of notifications.
-----
AWS update:

Increased Error Rates and Latencies

Oct 20 3:35 AM PDT The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now. Some requests may be throttled as they work toward full resolution.

The issue has been identified and is related to an ongoing AWS incident affecting multiple services: https://health.aws.amazon.com/health/status

Please continue to diversify your paging policies to incorporate Push and email.
-------------
Current AWS status:

Increased Error Rates and Latencies
Oct 20 2:27 AM PDT We are seeing significant signs of recovery. Most requests should now be succeeding. We continue to work through a backlog of queued requests. We will continue to provide additional information.

Oct 20 2:22 AM PDT We have applied initial mitigations and we are observing early signs of recovery for some impacted AWS Services. During this time, requests may continue to fail as we work toward full resolution. We recommend customers retry failed requests. While requests begin succeeding, there may be additional latency and some services will have a backlog of work to work through, which may take additional time to fully process. We will continue to provide updates as we have more information to share, or by 3:15 AM.

Please bear with us while we troubleshoot this issue — updates to follow.

Where possible, please adjust your personal paging policies to Push notification and email.

If you are affected by this reporting feature, please reach out to On-Call Support.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support