Stream API degraded availability

Last night we made some configuration changes to provide more stability to the cluster. We did not see peaks of errors or failed connections over the previous 12 hours.

We will follow up with a post-mortem later today.

The cluster is online again. We do anticipate a queue build up with our clients, so there could be a slight performance degradation in the upcoming minutes. We do not expect any failing connections to our system.

We continue to monitor the behavior of the cluster and our API. Tomorrow we will follow up with a post-mortem and action plan for improvements.

We have allocated the issue to a single node. After the recovery, we modified the configuration to be more stable. These updates are also applied to the other Nodes. We are running the final checks before the cluster can go online.

One of the nodes is in recovery; we are waiting till this is done before enabling the cluster again.

We are restarting nodes in our cluster and inspecting the reported errors.

In the meantime, all requests to our load balancer are blocked.

We will perform a complete cluster restart, expecting a brief full outage of the system.

We will stop traffic to our load balancer, temporarily rejecting all requests.

We're experiencing issues with our Stream API cluster and are seeing an increased number of failed connections.

Sorry for the inconvenience.