Elevated Error Rates for Log Queries and Monitors

This incident has been resolved.

Fix has been rolled out and we are currently monitoring to confirm full resolution.

We have successfully tested a fix for this issue and are currently deploying it to resolve this incident.

We're still working on a fix for historical data impacted by this incident.

We're still working on a fix for historical data impacted by this incident.

We're still working on a fix for historical data impacted by this incident.

We're still working on a fix for historical data impacted by this incident.

We're still working on a fix for historical data impacted by this incident.

We have deployed a fix and we are monitoring the results. We will provide another update once the issue is fully resolved.

At this time, newly ingested data is properly queryable, and monitors targeting Logs sent from 2023-10-03 20:40 UTC onwards are valid.

Queries targeting logs between 2023-10-02 11:40 UTC and 2023-10-03 20:40 UTC may return erroneous data. We are evaluating a fix that will restore query correctness for this time-window.

We have identified the underlying issue and are working on a fix.

We are continuing to investigate these issues, and will provide an update as soon as possible.

We are actively investigating issues with Log Queries returning unexpected results.
As a result of this issue, some users may experience issues querying logs on the web application or API, and with Logs based Monitors and Log-Based Metrics.