FireHydrant Outage History

FireHydrant experienced authentication issues with their Zoom integration that prevented the service from functioning properly for 2.4 hours. The team identified the root cause as refresh token problems and deployed a fix to restore normal integration functionality. The incident was resolved with monitoring in place to ensure the Zoom integration maintained healthy status.

FireHydrant experienced latency issues due to an advanced DDoS attack targeting their servers, lasting 6.9 hours. The attack involved bad actors attempting to add phone numbers to new accounts as part of a phone spam campaign. FireHydrant mitigated the issue by blocking the ability to add phone numbers to new accounts, which restored normal latency levels, though they continued working on a more permanent solution to safely restore the affected functionality.

FireHydrant experienced a minor service incident during a Slack workspace migration that lasted 1.3 hours. The incident was related to the migration process of their Slack workspace infrastructure. The issue has been resolved following the completion of the migration.

FireHydrant experienced a major incident where on-call schedules failed to load due to timezone discrepancies caused by a recent infrastructure upgrade. The issue affected the API endpoint responsible for listing on-call schedules, though the schedules themselves remained intact and paging functionality through Signals continued to work. The incident was resolved by rolling back the infrastructure changes, restoring full functionality after 1.2 hours.

FireHydrant experienced a major incident where Voice and WhatsApp notifications failed to be delivered to users. The service disruption lasted 23.1 hours before the issue was mitigated. The team implemented a fix and continued monitoring to ensure the notification systems remained stable.

FireHydrant's Zoom scribe was unable to join meetings for 68.9 hours due to invalidated credentials during a Zoom marketplace application migration, preventing AI transcription functionality on Zoom calls. The issue affected users' ability to automatically transcribe incident calls through Zoom integration, though Zoom meetings themselves continued to operate normally. The problem was resolved by updating Zoom credentials and reauthorizing failing connections, with Google Calendar integration recommended as a temporary workaround during the outage.

FireHydrant experienced a major incident where users were unable to declare incidents through Slack, which later expanded to affect the web UI, mobile app, and caused delays in alert acknowledgment and display. The incident lasted 1.8 hours, during which the team identified the root cause, implemented a fix, and worked through a backlog of delayed alerts to restore full functionality.

FireHydrant experienced an issue where the incident timeline interface failed to scroll properly, preventing users from viewing all timeline events. The problem was identified and a fix was deployed within 1.5 hours. This was classified as a minor service disruption affecting the incident management platform's timeline display functionality.