Datadog US1 Outage History

Datadog US1 experienced a major incident where Azure metrics could not be submitted, affecting the Metrics and Infra Monitoring components for 1.5 hours. The issue was identified and a fix was implemented, with the service returning to normal operation and Azure metrics reporting as expected.

Datadog US1 experienced increased latency when querying metrics, causing delays in loading dashboard and notebook graphs as well as delayed monitor evaluations. The issue affected the Monitors, Metrics and Infra Monitoring components for 1.4 hours. A fix was implemented and the incident was resolved after monitoring confirmed the solution was effective.

Datadog US1 experienced degraded performance affecting Database Monitoring and APM service pages for 38 minutes. Customers encountered failing database list pages showing zero databases, 503 errors when accessing database instance views, inability to interact with Database Monitoring data, and APM service page loading failures. The incident was resolved after deploying a fix and monitoring the results.

Datadog US1 experienced a 14-hour major incident with delayed monitor notifications and data processing issues across multiple services including Monitors, APM, Log Management, Metrics, CI Visibility, and RUM, caused by upstream AWS provider issues. The incident affected telemetry data ingestion from AWS and caused increased latency across the platform, though core metrics processing was gradually restored throughout the incident. The issue was fully resolved after AWS infrastructure problems were addressed.

Datadog US1 datacenter experienced delays in APM-based monitor notifications, causing delayed alert triggering and temporary monitor inconsistencies for users. The monitoring system was affected for 1.7 hours until the issue was identified and a fix was implemented. The service team deployed a resolution and continued monitoring to ensure stability.

Datadog US1 experienced elevated error rates for metrics queries, causing users to encounter errors when viewing metrics graphs in the web application or accessing metrics via API. The incident affected APM, Metrics, and Infrastructure Monitoring components for 35 minutes before being resolved with a deployed fix.

Datadog US1 experienced delays in ingesting AWS and Azure cloud integration metrics for 26 minutes. The incident affected Monitors, Metrics and Infra Monitoring components. The issue was identified and resolved with a fix implementation.

Datadog US1 experienced a 2-hour incident with increased latency in metrics processing, causing delays and gaps in metrics displayed on graphs and affecting the Metrics and Infra Monitoring components. To prevent false alerts from delayed data, affected monitors were automatically disabled during the incident and resumed normal operation once current data became available. The issue was resolved after the team addressed the lag in distribution metrics processing.

Datadog US1 experienced a major outage where notifications to multiple third-party services including PagerDuty, VictorOps, Slack, Webhook, Microsoft Teams, and OpsGenie failed to deliver, affecting the Monitors component. The issue was identified and fixed within approximately 1.5 hours, with new notifications resuming delivery, followed by a gradual catch-up process for delayed notifications that took an additional 4.6 hours to fully complete across all affected services.

Datadog US1 experienced increased latency in processing and storing APM traces, causing users to see missing or delayed traces in APM Trace Search and delays in trace-based monitors starting at 5pm UTC. The issue was identified and a fix was implemented within 36 minutes, with monitoring ongoing to ensure resolution.