FRA1 Events processing stopped

The platform is consuming events in real-time since 19:09 CET.
This incident has been resolved.

A good part of the backlog has already been processed.
The platform's detection is estimated to be back in real-time at 19:05 CET.
We will keep monitoring closely until that time.

The event processing is stable and we are catching up on lag slowly.
We can expect to be back in real-time in a couple hours due to the volume of events backlog that we have.
We will keep you updated on this.

We were able to mitigate the issue, our events processing pipeline is back up.
The platform is now consuming lag, we are monitoring this closely.

We lost several servers in a few minutes due to a network issue on our cloud-provider side.
Our event ingestion pipeline is safe from impact and we are still receiving every events.
However, our events processing pipeline is stopped since 15:23 CET, and we are taking lag on events processing and alerts raising.
We are currently reaching our cloud provider support to get more information in order to resolve this incident as fast as possible.