INC0129024

Current status: We've implemented the fix for this issue and monitored the environment to confirm that service was restored. Please note, customer recovery times may vary across the affected infrastructure. If you experience additional issues or have questions, please open a support case via Snowflake Community.

Customer experience: Customers hosted in the AWS - US East (N. Virginia) and AWS - US West (Oregon) regions experienced intermittent delays or timeouts while attempting to perform queries or use Snowsight. Starting at approximately 16:00 UTC, customers hosted in the AWS - US East (N. Virginia) region were unable to access or use Snowflake services and features.

Incident start time: 12:36 UTC March 19, 2025
Incident end time: 19:45 UTC March 19, 2025

Preliminary root cause: A code issue affecting an automatic scaling system triggered resource exhaustion and throttling on systems that process customer requests. After implementing an initial mitigation, a secondary, related issue was identified within a subset of the AWS - US East (N. Virginia) region that caused more severe impact to cloud services layer clusters that facilitate the scheduling of requests from virtual warehouses. The root cause of the secondary issue is currently under investigation.

A root cause analysis (RCA) document will be published within seven business days.

Current status: All clusters are reporting service health recovery after implementing mitigations for this issue. Please note, customer recovery times may vary across the affected infrastructure. We'll continue to monitor the environment until we're confident all services are functioning properly.

Customer experience: Customers hosted in the AWS - US East (N. Virginia) and AWS - US West (Oregon) regions experienced intermittent delays or timeouts while attempting to perform queries or use Snowsight. Starting at approximately 16:00 UTC, customers hosted in the AWS - US East (N. Virginia) region were unable to access or use Snowflake services and features.

Incident start time: 12:36 UTC March 19, 2025
Incident end time: 19:45 UTC March 19, 2025

Preliminary root cause: A code issue affecting an automatic scaling system triggered resource exhaustion and throttling on systems that process customer requests. After implementing an initial mitigation, a secondary, related issue was identified within a subset of the AWS - US East (N. Virginia) region that caused more severe impact to cloud services layer clusters that facilitate the scheduling of requests from virtual warehouses. The root cause of the secondary issue is currently under investigation.

Current status: After our additional mitigation efforts, service health is recovering. We're continuing to implement optimizations and monitor the affected environments while the remaining cloud services layer clusters fully recover. As such, customer recovery times may vary across the affected infrastructure. Since resolution is being observed in the region, failing over is no longer recommended for customers with replication enabled in other regions unless they are continuing to experience impact. We'll provide another update within 60 minutes.

Customer experience: Customers hosted in the AWS - US East (N. Virginia) region may be unable to access or use Snowflake services and features.

Incident start time: 12:36 UTC March 19, 2025

Current status: Within the AWS - US East (N. Virginia) region, we have identified an additional issue preventing the cloud services layer from processing warehouse requests as expected due to resource contention issues. We're continuing to implement multiple mitigations to help remediate impact including increasing capacity and, in parallel, reverting the most recent software releases in the region.

Customer experience: Customers hosted in the AWS - US East (N. Virginia) region may be unable to access or use Snowflake services and features. We recommend customers with replication enabled to other regions consider failing over outside of the AWS - US East (N. Virginia) region.

Incident start time: 12:36 UTC March 19, 2025

Current status: We identified an issue affecting an automatic scaling system, triggering resource exhaustion and throttling on systems that process customer requests. After implementing an initial mitigation, we have seen some customer recovery across affected deployments; however, the AWS - US East (N. Virginia) region is continuing to see impact to warehouse operations. We are attempting to implement additional mitigations to reduce resource contention issues within the region, and we'll provide another update within 60 minutes.

Customer experience: Customers hosted in the specified regions may experience delays or timeouts while attempting to perform queries or use Snowflake services and features via Snowsight. Queries may appear to be stuck in a running state. Customers hosted in the AWS - US East (N. Virginia) region may be unable to resume warehouses.

Incident start time: 12:36 UTC March 19, 2025

Current status: We've identified the source of the issue, and we're developing and implementing a fix to restore service. We'll provide another update within 60 minutes.

Customer experience: Customers hosted in the specified regions may experience intermittent delays or timeouts while attempting to perform queries or use Snowflake services and features via Snowsight. Queries may appear to be stuck in a running state.

Incident start time: 12:45 UTC March 19, 2025

Current status: We're investigating an issue with Snowflake Data Cloud. We'll provide an update within 60 minutes.

Customer experience: Customers hosted in the specified regions may experience intermittent delays or timeouts while attempting to perform queries or use Snowflake services and features.

Incident start time: 12:45 UTC March 19, 2025