Splunk On-Call is experiencing disruptions for Rules Engine functionality and Incident Timeline Data

This incident has been resolved.

We apologize for this inconvenience. Please reach out to On-Call Support with any questions:
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

The issue has been identified as upstream from the On-Call platform but impacting our CPU utilization. Mitigation is underway and we are now monitoring the results. Our engineering teams continue to work on this issue with urgency. We will provide updates as soon as we learn more information.

Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected.

Stay tuned for further updates as we work towards a complete resolution.

We apologize for this inconvenience. Please reach out to On-Call Support with any questions:
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected.

Impacted Functionality:
-Increased RegEx Rules Engine rule timeouts - Disabled Rules Engine Rules may be reenabled to resume functionality.

-UI and Mobile App Timeline data (incident cards) - Alert details may still be found via the On-Call API

-UI and Mobile App empty Incident alert details - Alert details may still be found via the On-Call API or our Reporting features: Incident Frequency, On-Call Review, and Response Metrics.

Please be advised that our engineering teams are working urgently on this issue. We will provide updates as soon as we learn more information.

We apologize for this inconvenience. Please reach out to On-Call Support with any questions:
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

We continue to investigate this issue. Please note that only a subset of instances are affected by this issue and no disruption to incident creation and notifications out to on-call end-users have been detected.

Apologies for any inconvenience.

Please reach out to On-Call Support with any questions:
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

We are continuing to investigate this issue.

Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected.

Impacted Functionality:
-Increased RegEx Rules Engine rule timeouts - Disabled Rules Engine Rules may be reenabled to resume functionality.

-UI and Mobile App Timeline data (incident cards) - Alert details may still be found via the On-Call API

-UI and Mobile App empty Incident alert details - Alert details may still be found via the On-Call API or our Reporting features: Incident Frequency, On-Call Review, and Response Metrics.

Please be advised that our engineering teams are working on this issue with urgency. We will provide updates as soon as we learn more information.

We apologize for this inconvenience. Please reach out to On-Call Support with any questions:
https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/