Aiven incident: Some Aiven Kafka clusters stopped reporting Prometheus metrics

Fix has been deployed and preventative maintenance will be scheduled for potentially impacted and impacted services.

We are still closely monitoring Aiven fleet of Kafka clusters. So far no new services were affected. We will continue to monitor and to work on a permanent fix.

We have applied a patch to all Aiven Kafka services which previously stopped emitting metrics to their Prometheus endpoints.

We are continuing to closely monitor affected services, but we are confident that the situation should now gradually get back to normal. In case of doubt, feel free to reach out to Aiven Support.

We have identified the root cause of the partial outage affecting Prometheus metric reporting. We also identified the 7 impacted services.

We prepared an Emergency procedure to patch those services. As we are applying this patch, we will also pro-actively reach out to impacted customers.

We will provide a further regular updates about this issue.

We are currently investigating a partial outage affecting Prometheus metric reporting of some Aiven Kafka clusters. As far as we can tell, Datadog users are not impacted.

We apologise for the inconvenience caused by this issue. We will provide regular update about our progress in resolving this issue.