Unresponsive shared hooks

This incident has been resolved.

We are continuing to monitor for any further issues.

The queue of messages has been fully processed and the system has returned back to normal operation. Shared hooks are now delivered without delays. We will now continue to monitor the systems and ensure that there is no regression.

We have implemented and validated a solution to the underlying problem. The queue of hooks is now being processed at an accelerated pace and we expect to return to normal operation in the following 90 minutes. The average hook processing delay is approximately 3 hours.

We have taken the necessary steps to stabilize the situation and started processing the queue of hooks at a faster pace. The main issue still persists and shared hooks are currently being processed with a significant delay.

Our engineers are currently pursuing two hypotheses with the goal to remediate the issue. We will post another update latest in an hour.

The issue still persists and shared hooks are being processed with a delay. The team is working towards addressing the root cause of the issue.

We have identified the issue and applied a fix. Currently shared hooks are responding but might be delayed.

We are currently experiencing issues with some of our services responsible for shared hooks on our clusters.
Shared hooks might currently be unresponsive. We are actively investigating the issue.