Investigating delayed session data ingestion on Cloud Dashboard

This incident has been resolved. Between 19:20 and 00:10 UTC, session data and observability ingestion was delayed by 45-60 min. The root cause has been addressed and data ingestion returned to normal by 00:10 UTC. No data was lost during this window.

mitigations have been deployed and we are now catching up to realtime.

the amount of data we are ingesting in the sessions view is outpacing our ability to ingest, thus causing a delay in ingestion. we are pursuing a few mitigations right now including moving to a larger database instance to relieve the processing bottleneck.

We are continuing to work on a fix for this issue.

Sessions data ingest remains delayed for about an hour. It's not impacting other dashboards.

we've identified the source of the slow ingest and are working on a mitigation. will update again in 30 mins

We are currently investigating delayed session data ingestion on Cloud Dashboard. No services appear to be impacted and we don't expect any data to be lost.