Partial production outage for function execution, event ingestion delay

The hotfix for event ingestion delay has been successful and the event ingestion backlog has been consumed. Events that are sent should trigger runs to be queued as normal. 98% of customers should be back to normal functionality.

Function execution still is affected for 2% of users. These users have been contacted. We are working with our infrastructure provider to increase capacity.

A hotfix has been deployed for the event ingestion backlog. New runs are being queued for execution. Data was not lost - the backlog will be consumed as fast as possible.

The customers affected with the queue issue are still affected. Again to confirm, if you have been affected, you have been contacted via Slack channel.

We are preparing a change to address the event ingestion backlog to reduce the number of customers impacted to those only experiencing the queue issue. More updates to follow.

We have pinpointed the issue to a queue that services a limited set of customers. Those users have been directly contacted. Additionally, the outage for this queue service is causing a backlog for new event ingestion. This is causing a large slow down and we're working into a solution to recover this issue now.

No event data has been lost.

We are actively investigating an issue with one of our queues that services a subset of customers. Not all customers are impacted. We are actively working to address this issue. We will provide further updates as we identify the cause and resolve the issue.