2025-03-20 Delays in Ingest

This incident has been resolved.

Dataset freshness has been restored to configured goals on customer tenants. We will continue to monitor freshness for the next 2 hours to ensure this incident has been resolved.

Recovery efforts are ongoing on the Observe side and we are getting closer to meeting data freshness goals on all customer datasets. We expect all remaining datasets freshness latency to be resolved within the next hour. We are continuing to monitor the systems very closely through the recovery process.

Our service provider, Snowflake, has completed changes to mitigate the networking issue on their end. https://status.snowflake.com/incidents/525rr3btnrz8.

Recovery efforts are ongoing on the Observe side and we are getting closer to meeting our customers' data freshness goals. ETA on dataset freshness catch up is approximately 1 hour. We are continue to monitor the systems very closely through recovery.

Snowflake has completed changes to mitigate the networking issue on their end. This is the incident they were working https://status.snowflake.com/incidents/525rr3btnrz8.

Recovery efforts are underway on the Observe side and we are steadily catching back up. ETA on restoration is approximately 1-2 hours to be fully restored and caught up.

We currently are monitoring the catch up and the current ingest of this region.

Snowflake is continuing to apply changes to mitigate the networking issue on their end. Recovery efforts are underway on the Observe side and we are ramping processing back up. https://status.snowflake.com/incidents/525rr3btnrz8. Both Observe and Snowflake have the highest level of focus and priority on resolving the issue. Currently all data ingest is continuing at a slow pace. Data acceleration is similarly delayed. We do not expect any data loss.

We are continuing our investigation with our service provider to mitigate the issue. Our service provider has identified a networking issue that is tracked by an incident https://status.snowflake.com/incidents/525rr3btnrz8. Snowflake is actively taking steps to mitigate the identified issue. There is no ETA yet on resolution. Both Observe and Snowflake have the highest level of focus and priority on resolving the issue. Currently all data ingest is continuing at a slow pace. Data acceleration is similarly delayed. We do not expect any data loss.

We are continuing our investigation with our service provider to mitigate the issue. Our service provider has identified an issue that is affecting more than 3 customers that is tracked by an incident https://status.snowflake.com/incidents/525rr3btnrz8. Both Observe and Snowflake have the highest level of focus and priority on resolving the issue. Currently all data ingest is continuing at a slow pace. Data acceleration is similarly delayed. We do not expect any data loss.

We are continuing our investigation with our service provider to mitigate the issue. Our service provider has identified an issue that is affecting more than 3 customers. Currently all data ingest is continuing at a slow pace. Data acceleration is similarly delayed. We do not expect any data loss.

We are continuing our investigation with our service provider Snowflake to try to identify the root cause of the issue and mitigate. Currently all data ingest is continuing but loading of data is failing. We do not expect any data loss.

We are continuing our investigation with our service provider Snowflake to try to identify the root cause of the issue and mitigate. Currently all data ingest is continuing but loading of data is failing. We do not expect any data loss.

We are continuing our investigation with our service provider Snowflake to try to identify the root cause of the issue and mitigate. Currently all data ingest is continuing but loading of data is failing. We do not expect any data loss.

We are currently still investigating and actively working the incident with our service provider Snowflake to try to identify the root cause of the issue and mitigate. Currently all data ingest is continuing but loading of data is failing. We do not expect any data loss.

We have identified that the issue is being caused by an issue with our service provider Snowflake. Currently all ingest is failing but we do not expect any data loss. Our engineers are actively working with Snowflake to try to identify the root cause of the issue and mitigate. We will post further updates as they become available.

We’ve identified an issue causing delays in ingest in the following regions: US East (N. Virginia). As a result, some users may experience see delays in ingesting data. Our team is actively investigating the root cause and working to restore normal performance as quickly as possible. We’ll provide updates as we learn more. Thank you for your patience!