Expel Outage History

Expel experienced a major incident with their customer device secrets service starting at 2pm EDT, preventing their infrastructure from accessing customer security devices and disrupting alert ingestion. The issue was resolved after 1.2 hours with a fix implemented around 3:09pm EDT, restoring full system availability. All customer security devices returned to healthy status and resumed ingesting alerts with no data loss.

Expel's notification system experienced delays starting at 1pm ET, affecting email and Slack notifications to customers. The issue was resolved after 28 minutes, with all backlogged notifications delivered and normal service restored.

A change to Expel's device credentials storage subsystem caused invalid credentials for a subset of security devices, making them unhealthy and preventing alert ingestion for approximately one week before detection. The incident affected alert ingestion capabilities for the impacted devices, creating a backlog of unprocessed security alerts. Expel identified the affected devices, restored their credentials, and confirmed that delayed alerts were ingested once the devices returned to a healthy state.

Expel's Workbench was unable to deliver notifications to Microsoft Teams for 4.3 hours starting at 4pm UTC due to a Microsoft Teams service issue affecting bot messaging and delivery. The incident was caused by Microsoft infrastructure problems, not issues with Expel's systems, and primarily impacted users in the West United States region. Microsoft resolved the underlying service degradation and reprocessed all delayed messages, restoring normal notification delivery to Teams.

An AWS GuardDuty service outage caused Expel's integration with GuardDuty to become unhealthy, disrupting alert ingestion for 2.1 hours. The issue was resolved by implementing a workaround that bypassed the faulty AWS region, allowing alert ingestion to resume from healthy regions.