Errors accessing log data in Knock Dashboard

We have caught up on all delayed log writes accrued. We are continue to monitor our cluster closely.

We are beginning to see recovery. Reads of log data in the Dashboard and V1 API are succeeding again. We are beginning to catch up on the backlog of failed writes that have accrued in our system.

The root cause here is an outage with our log datastore provider, ClickHouse Cloud. Outage details are available on their status page: https://statuspage.incident.io/clickhousecloud/incidents/01K6AZ5P1XPDSX7696YC1SDVKG.

We are seeing a full outage of both reads and writes of log data. This outage is impacting message events, workflow run logs, and message delivery logs.

At this time workflow execution and notification delivery remains unaffected.

We currently believe this to be caused by a vendor outage and we are working with our vendor to resolve.

We’re investigating increased rate of errors writing log data and retrieving in the Dashboard. Workflow execution and notification delivery is unaffected.