API failures across multiple clusters and degraded webApp performance

We have fully recovered from this incident and are marking it resolved. If you have any further questions please reach out to support@iterable.com

As of now we have completely caught up on ingestion lag with all services returning to normal. We will continue to monitor performance with that next update at 12 PM PT or sooner.

We are continuing to monitor for any further issues.

Web app and API endpoints have completely recovered at this point. However, there are still a subset of customers that may be experiencing an ingestion lag that is currently draining. These customers may still be seeing delays in user updates, event calls, and event triggered journeys. We are continuing to monitor this and will provide our next update at 11 AM PT or sooner.

Our Engineers have identified the root cause of the issue and are actively working on deploying a fix. Currently our API endpoints have recovered, but in the meantime, customers may still be experiencing slowness in scheduled and triggered Journeys, scheduled and triggered campaign sends, custom events, user updates, and more. Customers may also be experiencing webApp performance degradation as well including segmentation, list uploads, and viewing campaign details. While the fix is being deployed, to clarify, this issue is specifically impacting All clusters numbered 100+. Our next update will be at 10:00 AM PT or sooner.

We are continuing to investigate this issue.

Beginning around 6:20 AM PST we were alerted to a spike in API errors across multiple endpoints impacting a number of specific customer clusters. All clusters numbered 100+ may be experiencing a spike in 5xx API errors across all endpoints. This may impact areas of the app such as scheduled and triggered Journeys, scheduled and triggered campaign sends, custom events, user updates, and more. Customers may also be experiencing webApp performance degradation as well including segmentation, list uploads, and viewing campaign details.

Our engineering team is continuing work on identifying the underlying cause and exploring remediation options. Next update will be at 9 AM PST or sooner. If you have questions please reach out to support@iterable.com