Compute Backup Service Issue

At this time the issue that affected the Backup service has been fixed, and we will now consider this incident resolved. If you continue to experience problems, please open a ticket with our Support Team.

It has been determined that a small number of customers is still unable to make backups. As such, we are returning this incident to the "Identified" status. We expect that the issue will be resolved by tomorrow, January 23rd.

At this time we have been able to correct the issues affecting the Backup Service. We will be monitoring this to ensure that it remains stable. If you continue to experience problems, please open a Support ticket for assistance.

Upon further investigation, our team has discovered a wider scope of impact for this issue. For any backup that was affected by this, the purging of files of 500 MB or larger impacted all available backups, instead of just backups that ran during the earlier mentioned window of 9:00 UTC on January 19, 2024 to 03:00 UTC on January 20, 2024. As a result of this discovery, our team will be making backups from affected instances unavailable to the customer. This is intended to prevent restores from backups with bad integrity.

Our team has deployed a fix to address any affected backup to address the ability to create new backups. They expect this fix to take up to 48 hours to address all affected backups. We will update this posting when the fix has been implemented.

As a matter of caution, we continue to recommend any restores needed from the Backup Service be deployed to a new compute instance. Should the restore be successful, you can then transfer the IP address between the old and new compute instance. This will alleviate any DNS changes needed and the only downtime should be a result of rebooting the instance.

If you have already used an affected backup to overwrite your instance’s disks, we recommend utilizing the Recovery Image that was created when the compute instance’s disks were deleted. You can deploy an Image to an existing compute instance to recover from any files that were not included in your restore but were present on the instance at the time the disks were deleted.

In the event that there is data from an affected backup that a customer requires, please open a Support ticket for assistance. We can make these affected backups available to you for the next fourteen days.

We are continuing to work on a fix for this issue.

Our subject matter experts are continuing to work to address this issue. At this time we have identified that customers whose backup service ran during this window may have experienced data loss. Files of more than 500MB were permanently purged from backups and snapshots and cannot be recovered. Our subject matter experts are continuing to work on restoring the backup operations. However, some customers may experience backup failures while the issue is being mitigated. Due to the nature of this backup failure, we suggest any backup restores are restored to new Linodes to ensure your backups contain the data you require without removing the existing disks of your Linode.

A recent software release contained a defect that has impacted the integrity of backups and snapshots captured between around 19:00 UTC on January 19, 2024 and 03:00 UTC on January 20, 2024. This change has been rolled back and our subject matter experts are investigating further. Thank you for your patience and understanding.