Webhook Processing Delays

Realtime webhook processing has been fully restored, and has maintained stability after monitoring. All new webhook events are now completing in realtime as expected. To further strengthen system stability, the engineering team has deployed an additional mitigation to prevent webhook queue backlogging under similar spike conditions.

A thorough postmortem will be completed a Root Cause Analysis (RCA) can be shared with customers - please email support@alloy.com if you would like to receive the RCA. We are also investing in further improvements to the reliability and resiliency of our webhook queueing mechanism in the future to prevent event loss in the future.

Please note, action is required to ensure applications process:

During the incident, webhook events generated between 2:24 PM - 4:38 PM ET and 5:42 - 5:45 ET were not successfully processed and were permanently lost. Webhooks generated outside of this time window were not impacted.

This issue affects applications that are stuck at a Journey step dependent on an Alloy webhook response to proceed (for example, action nodes), as well as applications that completed with a Manual Review outcome but are not reflecting back in core after review.

Applications submitted during the affected window that are currently blocked due to a missing webhook dependency will need to be replayed. We recommend resubmitting those applications via API. Re-running an application will leverage cached data by default and will not trigger new calls to external data vendors.

Realtime webhook processing has been fully restored. All new webhook events are now completing in realtime as expected.

During the incident, webhook events generated between 2:24 PM - 4:38 PM ET and 5:42 - 5:45 ET were not successfully processed and were permanently lost. Webhooks generated outside of this time window were not impacted.

This issue affects applications that are stuck at a Journey step dependent on an Alloy webhook response to proceed (for example, action nodes), as well as applications that completed with a Manual Review outcome but are not reflecting back in core after review.

Applications submitted during the affected window that are currently blocked due to a missing webhook dependency will need to be replayed. We recommend resubmitting those applications via API. Re-running an application will leverage cached data by default and will not trigger new calls to external data vendors.

A thorough postmortem will be completed a Root Cause Analysis (RCA) can be shared with customers - please email support@alloy.com if you would like to receive the RCA. We are also investing in further improvements to the reliability and resiliency of our webhook queueing mechanism in the future to prevent event loss in the future.

We are currently experiencing delays in realtime webhook processing. The root cause has been identified and we are applying mitigation measures. We are now monitoring recovery as the processing queue returns to normal levels.

Webhook events continue to be accepted; however, processing and delivery responses may be delayed until the existing backlog has been fully cleared.