Outage in Datadog US1

Delayed Processing for a Subset of Metrics

Resolved Minor
August 01, 2023 - Started over 2 years ago - Lasted about 9 hours
Official incident page

Incident Report

We are investigating increased latency processing Processing for Distribution Metrics coming from the agent. As a result of this issue, some users may see delays or gaps for a subset of their metrics on graphs. RUM/Synthetics/Logs/Processes measures that depend on distribution metrics will also see delays. To prevent spurious alerts, we have temporarily disabled monitors based on this data.
Components affected
Datadog US1 Monitors Datadog US1 Log Management Datadog US1 Metrics and Infra Monitoring Datadog US1 Synthetics

Need to monitor Datadog US1 outages?

One place to monitor all your cloud vendors. Get instant alerts when an outage is detected.

Start Free Trial
Latest Updates ( sorted recent to last )
RESOLVED over 2 years ago - at 08/02/2023 05:45AM

This incident has been resolved.

MONITORING over 2 years ago - at 08/02/2023 03:08AM

Distribution metrics lag has been resolved. As a result of the issue, some users may see gaps for distribution metrics on graphs. Monitors evaluation is now re-enabled for monitors based on this data.

MONITORING over 2 years ago - at 08/02/2023 12:43AM

We continue to make progress on processing the lag on distribution metrics and are monitoring any further issues.

MONITORING over 2 years ago - at 08/01/2023 10:57PM

We continue to process the lag on distribution metrics

MONITORING over 2 years ago - at 08/01/2023 09:42PM

A fix has been implemented and we are monitoring the results.

IDENTIFIED over 2 years ago - at 08/01/2023 09:37PM

The issue has been identified and a fix is being implemented.

INVESTIGATING over 2 years ago - at 08/01/2023 09:37PM

We are continuing to investigate this issue.

INVESTIGATING over 2 years ago - at 08/01/2023 09:07PM

We are continuing to investigate this issue.

INVESTIGATING over 2 years ago - at 08/01/2023 09:05PM

We are investigating increased latency processing Processing for Distribution Metrics coming from the agent.
As a result of this issue, some users may see delays or gaps for a subset of their metrics on graphs.
RUM/Synthetics/Logs/Processes measures that depend on distribution metrics will also see delays.
To prevent spurious alerts, we have temporarily disabled monitors based on this data.

Latest Datadog US1 outages

Delayed Processes data - 1 day ago
Delayed APM metric ingestion - 1 day ago
Metrics data ingestion delayed and monitor evaluations degraded - 4 days ago
Web Application Not Loading - 24 days ago
Delayed Monitors Notifications - 26 days ago

Status Aggregator for All Your Third-Party Services

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4600 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook