Issue with Runbooks Attaching to Incidents

FireHydrant experienced a prolonged delay of runbook executions due to a sudden outlier amount of traffic to our services. This traffic caused a shared worker queue to flood and go into a "spin lock" preventing other work from executing, such as runbook attachment. Existing incidents were not impacted from this incident, and our systems have been scaled to handle traffic spikes like the one experienced in the future.

All backlogs have successfully cleared and Runbooks are functioning as expected

We've processed the backlog of Runbook attachments and are seeing Runbooks execute as expected

We've identified a backlog of runbook attachments and are working to process them

We are continuing to investigate an issue with runbooks not attaching to incidents automatically, and manually attached runbooks not executing

We're investigating an issue with Runbooks failing to attach to incidents

We are currently investigating this issue.