Restoration of historical data in AWS, US-East-1

This incident has been resolved

We sincerely apologize for the length of this incident. Restoration of data from February 2 onwards is now substantially complete, and the second phase of restoration, covering data prior to February 2, is underway, and has already been completed for many of our customers.  While we continue to restore the remaining historical data, some customers may notice elevated time-to-become-readable (TTBR) on queries.  We apologize in advance for the inconvenience that this may cause.  We are trying to maintain the responsiveness of the service while also restoring the last of the historical data as quickly as possible. Please contact support with any questions or concerns.

We apologize for the length of time that it has taken to restore customer data that was inadvertently deleted.  The data is all available in our backups, but it is taking us an unacceptably long time to restore it. as we have had to balance the restoration of historical data with service availability of the cluster. After we have resolved this issue, we will be revisiting our backup/restore strategy to improve the performance of cluster-wide restorations.  For this particular restoration, we have been restoring the data in two phases (data from February 2-9, and data prior to February 2nd).  Most customers should now be able to see all of their data from February 2nd onwards, with the last remaining data from this time range expected to be available by Sunday February 20th.  Once we have restored all the data from February 2nd onwards, we will be able to give a more accurate time estimate to restore the remaining older data.  Once again, we sincerely apologize for how long it is taking us to restore all of your data.  After we have completed the restoration, we will provide a full RCA for this outage.

Our team continues to actively work to restore data for all affected users.

We apologize for the ongoing disruption to prod101-us-east-1. We are continuing to restore data from our backups. During the restore, there may occasionally be heightened TTBR.

Our team has been working around the clock to restore data availability for all affected users. Work will continue through the weekend with no interruption. We do not yet have a resolution timeline to share, but we are committed to providing updates as soon as they are available until data restoration is complete. We sincerely apologize for the continued impact.

Our team continues to actively work to restore data for all affected users.

Our team continues to actively work to restore data for all affected users.

Our team continues to actively work to restore data for all affected users.

Our team continues to actively work to restore data for all affected users.

Our team continues to actively work to restore data for all affected users.

Our team continues to actively work to restore data for all affected users.

Our team is actively working to restore data for affected users.

Our team has identified the issue and is actively working to restore data for affected users.

Our team has identified the issue and is actively working to restore data for affected users.

Our team has identified the issue and is actively working to restore data for affected users.

We are continuing to work on the fix for this issue.

We are continuing to work on the fix for this issue.

We are continuing to work on the fix for this issue.

The issue has been identified and a fix is being implemented.

We are investigating an issue related to queries. Some recent data is not showing up on query responses