Production Ingest Pipeline Degradation

All systems are operational and working as expected.

All services are operational and we are monitoring for any further issues.

We have re-enabled all services, webhook sending, and the display of alerts/events should be processing again.
We are monitoring for any further issues.

We have tracked the cause of the problem and are investigating to prevent it from happening again in the near future. Sending webhooks have been temporarily disabled as we investigate, but will still be sent once we are fully operational.

Data is still being processed successfully, but there may be a delay in receiving alert/event webhooks or assets/alerts/events showing in the console or API.

We are currently recovering. Some systems may still be slow or partially inoperable, such as updating webhooks, receiving webhooks, and alerts/events showing in the console.

Team is continuing to take actions to restore operations. Until then, event ingestion, webhooks/alerting, new device registration, and some API operations in our US region will be severally degraded.

Cloud detections for our ransomware protections, Atlas API, are still online.

We are continuing to work on a fix for this issue.

Root cause has been identified and the team is working to address the problem.

Halcyon has identified performance degradation in the ingest processing pipeline and is investigating