Degradation across most services

From 11:25 PM PDT Sept 18, 2023 until 4:45 PM PDT Sept 19, 2023, our customers experienced high latency of up to 12 hours in all our products and services. This issue started by an AWS networking incident that disabled all our services except our data reception and lasted until 11:05 PM on Sept 18th. From that time, we worked on processing our 12 hours of backlog which were fully cleared by 4:35 PM Sept 19th. All our services are working normally and there was no data loss during this incident. Further actions are not required from our customers.

As of now, about 99% of realtime data are showing up in charts and dashboards. Full recovery of data is expected to be before 4:30 PM PDT including data warehouse exports.

As of now, about 85% of realtime data are showing up in charts and dashboards. Full recovery of data is expected to be before 4:30 PM PDT, and data warehouse exports will catch up before 5:30 PM PDT.

As of now, about 70% of realtime data are showing up in charts and dashboards. Full recovery of data is expected to be before 7:30 PM where data warehouse exports will catch up before 8:30 PM.

We are still working on speeding up the rate of recovery. However, our current estimated time will be 18 PDT for all services.
We will update the estimated times every other hour.

The underlying problem has been fixed, and our systems are working through the backlog. Customers should not see any data loss from this issue
- We expect to be fully caught up with ingestion by 16:00 PT
- Export systems will recover shortly after ingestion as the system catches up
- Reporting systems are expected to be fully caught up by 18:00 PT. in the meantime, customers may not see recent data in their charts/dashboards

We will give updates throughout the day as our systems make progress

All our components and services are working as expected. We are keeping this incident open and our on-call team are monitoring the systems to make sure there will be no surprises during the night.
We will resolve this incident after 6-8 hours of normal execution.

As of 11:25 PM PDT Sept 18, 2023, the AWS incident is resolved and we are resuming our components and services. There will be a large latency for the data to get processed and we expect to catch up in the next 10 hours. Most other services are recovered. Please see the updated list of affected services below.

AWS issue is not resolved. We are still monitoring the status and trying to find a workaround using a few AWS services that have recovered.

AWS Services started to recover but not completely. We are still monitoring and waiting for AWS to resolve their outage.

We are still waiting for AWS to resolve their networking issue.

We are still waiting for AWS to resolve their networking issue.

We are still waiting for AWS to resolve their networking issue. We will update here as soon as AWS problem is resolved.

On September 18 at 11:43am PDT, AWS identified a networking issue in their cloud and are working to mitigate the issue.

At this time, we do not see any impact to data reception. However, most other services are affected. We do not expect any data loss. No action is required from our customers at this time.

We are continuing to work on a fix for this issue.

On September 18 at 11:43am PDT, AWS identified a networking issue in their cloud and are working to mitigate the issue.

At this time, we do not see any impact to data reception. However, most other services are affected. We do not expect any data loss. No action is required from our customers at this time.