[US Region] Treasure Workflow - Partial Outage in Workflow Service

Our Workflow service had an outage from 10:50 am PST on 25th Sep. 2023. From that time, the workflow requests went to pending status. We fixed the incident and deployed our fix at 2:40 pm PST on 25th Sep. 2023. During this outage time window, the customer workflows might experience some delays. After fix deployment, the Workflow service is working as normal, so the service started to resume pending workflows while handling new requests as well.

The incident has been resolved.

We still have 20% of pending workflows to catch up. The remaining pending workflows will be processed within 30 minutes. We are continuing to monitor for any further issues.

The half of pending workflows are processed without any issue. The remaining pending workflows will be processed within an hour. We are continuing to monitor for any further issues.

The pending workflows are resuming now, but it will be taking 1-2 hours to backfill all pending workflows. We are continuing to monitor for any further issues.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We have observed the issue that the workflows are pending with partial outage in service. We are currently investigating the issue.

We have observed the issue that the custom scripts execution failure. We are currently investigating the issue.