Spike in Pipedream Internal Errors for workflows

All systems are operational. Incoming events should be processed successfully. We've also processed the backlog of events that arrived during the first part of the incident, from ~1:00am UTC to 3:51 UTC.

From 3:51 AM UTC to 6:04 AM, we had to disable incoming events due to some of the load issues we were experiencing. Events sent during this time may be retried by the source services. At 6:04am, we re-enabled the collection of incoming events again.

We'll follow up with a detailed retrospective of this incident as soon as possible.

AWS has shipped a fix for the issue. We're restarting our services and bringing workflows back online. We'll send another update as soon as that's done.

AWS is still working on a patch for the issue. We're still in active communication with them.

The AWS Lambda team confirmed the issue was due to the scale of volume Pipedream is running on Lambda. They're working a fix now. We'll update this incident again soon.

We are continuing to work on a fix for this issue.

AWS has escalated to more teams internally. The issue is still ongoing.

We're continuing to discuss this with the AWS team. The issue is still ongoing.

AWS Lambda — part of the service we use to run workflows — has identified a service issue. They're working on it, and we'll communicate updates here.

We're seeing a spike in Pipedream Internal Errors across workflows, and we're investigating.