Delayed scheduling and execution

The teams latest fixes are live and we're observing reduced delays that gradually have improved between 16:00 UTC and 18:00 UTC. We're continuing to work on additional fixes and optimizations to improve performance.

We have identified an issue with core execution logic related to flushing data for history and traces that may cause a delay in our system under certain circumstances. We are working on a fix to optimize this to reduce the overhead of this operation.

Additional database optimizations have been released which has reduced latency on function execution to normal levels. We continue to monitor and have several further optimizations that we'll release throughout the coming days.

There is an ongoing investigation into function execution latency.

The incident is now resolved and the system is full operational. There are not steps required to recover as no data loss or issues should have been created other than system latency.

We have deployed an improvement to optimize our database indexes resulting in a significant performance increase. The system is performing as expected and we continue to monitor the system.

We have identified the cause of the issue. We're actively working on implementing a fix to resume normal operation of the system.

We are actively investigating an issue with scheduling delays and execution, as well as dashboard performance. We will provider further updates as we identify the cause and resolve the issue.
No impact on event ingestion confirmed at the moment.