Data Processing Delays

This incident has been resolved.

AWS has resolved the underlying issues. Our data ingestion system in the EU environment has fully recovered, and the US environment is currently in the process of recovering. We will continue to closely monitor the system until full stability is confirmed.

We have identified the root cause as an AWS dependency that is experiencing errors in both EU and US production environments. We are currently waiting for AWS to resolve the underlying issues and will continue to closely monitor the situation.

We've identified the root cause as being a critical dependency in AWS experiencing a high error rate. We are monitoring and communicating with AWS to resolve this as soon as possible.

Our data processing systems are delayed. This incident started at 6:10pm PDT.

Current status:
a) Our ingestion systems are working as expected and no data is lost. Our processing systems are not processing newly ingested data.
c) Impacted customers will see delayed metrics from /batch endpoint for the last 80 minutes and metrics from real time endpoint for the last 33 minutes.

We are investigating the issue and will post an update in an hour or earlier if we identify the issue.