Datadog US1 Outage History

Datadog US1 experienced a 2-hour incident with increased latency in metrics processing, causing delays and gaps in metrics displayed on graphs and affecting the Metrics and Infra Monitoring components. To prevent false alerts from delayed data, affected monitors were automatically disabled during the incident and resumed normal operation once current data became available. The issue was resolved after the team addressed the lag in distribution metrics processing.

Datadog US1 experienced a major outage where notifications to multiple third-party services including PagerDuty, VictorOps, Slack, Webhook, Microsoft Teams, and OpsGenie failed to deliver, affecting the Monitors component. The issue was identified and fixed within approximately 1.5 hours, with new notifications resuming delivery, followed by a gradual catch-up process for delayed notifications that took an additional 4.6 hours to fully complete across all affected services.

Datadog US1 experienced increased latency in processing and storing APM traces, causing users to see missing or delayed traces in APM Trace Search and delays in trace-based monitors starting at 5pm UTC. The issue was identified and a fix was implemented within 36 minutes, with monitoring ongoing to ensure resolution.

Datadog US1 experienced a major incident where monitor notifications were significantly delayed for 5.6 hours starting at 12:00 UTC. The issue affected the Monitors component, preventing users from receiving timely alerts about their infrastructure and application monitoring. The incident was resolved after engineers identified the root cause and implemented a fix, with monitoring confirming the resolution.

Datadog US1 experienced an issue where metric monitors, primarily those based on distribution metrics, intermittently reported "No Data" status, making monitor alerting unreliable. The incident affected the Monitors component and lasted 2.1 hours. The issue was resolved after implementing a fix and monitoring the results.

Datadog US1 experienced a major incident where users encountered errors when trying to load the web application. The web application component was affected for 1.1 hours, preventing some users from accessing the service. The issue was identified and resolved with a fix that restored normal functionality.

Datadog US1 experienced increased latency in processing RUM (Real User Monitoring) data for 2.7 hours. Users encountered gaps or delays in RUM graphs and empty or partial query results across RUM Sessions, RUM Analytics, RUM Application, and Error Tracking pages. The underlying issue was identified and a fix was deployed, with no data loss reported and backfilling planned once service was fully restored.