Datadog US3 Outage History

Datadog US3 experienced increased latency across multiple products including RUM, App Builder, Audit Trail, APM, Metrics and Infra Monitoring, and Monitors, causing delays in data across the platform for users. The incident was caused by an underlying Azure infrastructure event and lasted 11.9 hours. All services were fully recovered with APM requiring additional time for trace backfilling, and the incident was resolved while monitoring continued due to the ongoing infrastructure event.

Datadog US3 experienced delays in Metrics and Distributed Metrics processing that caused Monitor evaluations to be skipped. The incident affected Metrics and Infra Monitoring along with Monitors functionality for 1.1 hours. Mitigation was implemented and the service fully recovered.

Datadog US3 experienced a major incident where Azure metrics could not be submitted, affecting the Metrics and Infra Monitoring components for 1.4 hours. The issue was identified and a fix was implemented, with the service returning to normal operation and Azure metrics reporting as expected.

Datadog US3 experienced failing queries for Cloud Networking and Service Checks that affected historical data older than April 29th 13:00 UTC, while recent data queries remained unaffected. The Cloud Network Monitoring component was impacted during this incident. The issue was identified, fixed, and resolved after 3.1 hours of service disruption.

Datadog US3 experienced elevated error rates and increased latency in processing Service Check and Cloud Network Monitoring, causing users to see gaps, delays, or partial query results in their monitors. The incident lasted 32 minutes, with the underlying issue identified and a fix implemented. The service was fully restored with no data loss reported.

Datadog US3 experienced increased latency in log processing for 36 minutes, causing delays and gaps in log query results for some users. The Log Management component was affected, and monitors related to delayed data were temporarily disabled to prevent false alerts. The underlying issue was identified and resolved with a deployed fix, with all delayed data being backfilled once service was restored.

Datadog US3 experienced increased latency across multiple products including APM, Log Management, Metrics and Infrastructure Monitoring, RUM, Database Monitoring, and Agent Repository, causing delays in data visibility across the platform. The latency issues affected users for approximately 1 hour before returning to normal levels. The incident was fully resolved after continued monitoring confirmed system stability.