Cypress Cloud Outage

We are observing normal operations and performance, and mitigations that put into effect earlier have served their purposes and are now unnecessary and thusly rolled off.

Services have reached normal operational levels. We are still seeing slowness in the same services cited as still-recovering in the broader aws outage (e.g., lambda and compute provisioning). Some Cypress Cloud functions are working through backlogs and remain lagged (they are noted as degrated).

Setting incident status to `monitoring`.

We are continuing to onramp additional infrastructure; however, we are not yet at our default levels of activity in all places. Upgrading to `monitoring` to communicate the improvement. Note that applicaiton quality services are still significantly lagging other services, but we are ramping up infrastructure there as well.

Recording services and analytics are normal
UI is at normal levels of activity + supply
Billing and Integrations still have some third party dependencies which are recovering
Application quality is significantly lagged

Some of the key bottlenecks in our recovery process on the AWS side of the infrastructure are beginning to show significant improvement. We have tightened some volumes in order to help with recovery throughput, which results in intermittency with the affected services. In other words, we are still recovering and are not yet at usual activity volume. As more resources are regained, we will be improving throughput into services as well.

We have some initial mitigation in place for the Cypress Cloud UI. We are still experiencing degraded service and underlying outages; we are continuing to take mitigation steps.

We are continuing to work on a fix for this issue.

The ongoing AWS outage in US-EAST-1 region have impacted several services that we rely on, especially around message routing and cluster provisioning.

We continue to see an outage with our Dashboard. Application quality services are operational, but experiencing significant processing delays Services that handle recordings and other critical areas of Cypress Cloud are operating normally.

The team continues to attempt mitigation strategies as we work towards recovering remaining services as quickly as we can.

We have confirmed Test Recordings to Cypress Cloud are operating normally. UI Coverage and Accessibility processing for recorded runs are operational, but experiencing elevated latencies due to limited compute capacity. Dashboard continues to experience a major outage and we are actively working on resolving the same.

AWS Services are experiencing network connectivity issues in the US-EAST-1 region. While there are early signs of recovery, multiple Cypress Cloud services continue to experience outages and we are actively pursuing parallel approaches to restore capacity.

Flagging incident as identified; recovery still in progress with resource constraints impacting services restoration

AWS services are incrementally recovering; but compute services are still experiencing issues & capacity remains constrained, impacting service restoration.

AWS Operational Issue affecting multiple services in N. Virginia (https://health.aws.amazon.com/health/status?path=open-issues) identified as root cause

We are currently investigating this issue.