Provider outage

All the delayed events were fully handled at 8h50.

We're continuing to monitor the processing closely and are seeing steady progress in reducing the backlog. While it may still take some time to fully catch up, we’re doing everything we can to maintain stability and ensure no data is missed.

The backlog of the queued events is still being processed at maximum capacity. Our team is dedicated to clearing this backlog as efficiently as possible, ensuring that all events are handled promptly.

Cluster recovery is done with no data loss. We are still processing the backlog of queued events, at maximum capacity.

We are making great progress on fixing the event storage cluster.
All "cold" data is progressively getting available.
For all events ingested after 03:30 CEST, we are processing the delayed events.
We found a way to speed up the recovery process. The event storage cluster is steadily recovering.

We are still making progress on the event storage cluster.
All "cold" data is progressively getting available.
For all events ingested after 03:30 CEST, we are processing the delayed events.
The situation is progressively recovering however we had to slow down the process for the moment due a very high number of parallel tasks causing a risk for the cluster. We are trying to find ways to improve the situation faster.

We are still making progress on the event storage cluster.
All "cold" data is progressively getting available.
For all events ingested after 03:30 CEST, we are processing the delayed events.
The situation is progressively recovering however we had to slow down the process for the moment due a very high number of parallel tasks causing a risk for the cluster. We are trying to find ways to improve the situation faster.

We are still making progress on the event storage cluster.
All "cold" data is progressively getting available.
For all events ingested after 03:30 CEST, we are processing the delayed events.
The situation is progressively recovering.

We are still making progress on the event storage cluster.
All "cold" data is progressively getting available.
For all events ingested after 03:30 CEST, we are currently fixing the situation.

We are still making progress on the event storage cluster.
All "cold" data is progressively getting available.
For all events ingested after 03:30 CEST, we are currently fixing the situation.

We are still working on the event storage cluster.
We are also making progress with alerts without events. Events linked to alerts are progressively available in the event storage cluster.

We are still working on the event storage cluster.
So far, event search are working but the oldest events are not available for search.
We are fixing this situation progressively meaning more and more older events will be available later.
On another hand, all events ingested from this morning 03:30 CEST are not available in the event storage cluster.

We are still working to stabilize the event storage cluster.
So far some event query and search are working however all data are not available for the moment.

We are still preparing a fix to rollout on our whole event storage cluster.
In the mean time, we fixed the automation cluster.

Most services are up.
There is still some issues with our event storage cluster making events and events search not being available.
All events are still received and properly processed.
On another hand, automation (playbooks) is also having issues.
We are working quickly to fix these situations.

We had an outage on our main provider, and the network went down.
We are currently recovering access to the platform and fixing the different issues.