Missing or outdated last seen data

All metrics have been stable for some time now. Last seen data is being served as normal and all service is fully restored. This concludes the incident. Thank you for your patience and understanding.

Last seen data is now back to normal. Updates are being stored, and the information is read back for the appropriate requests. We are doing some final checks and monitoring before declaring the incident as resolved.

Engineers are still in the process of reading last seen data from the new cluster, but last seen data is now being written.

The new cluster is online; engineers are slowly ramping up traffic to it and verifying correctness and load.

Unfortunately in the process of trying to recover last seen data during the previous incident, the cluster was discovered not to be able to handle the load.

Engineers have worked to bring last seen data into a brand new cluster with increased capacity that will be sufficiently stable for the longer term, and will soon be able to begin using it for last seen data reads and writes.

We are currently working to make the cluster that holds last seen data fully operational.

We are continuing to investigate this issue.

Last seen data may be missing from some responses and outdated in others. Our engineers are working on the cluster that stores this data to recover this functionality fully.