Virtualizations are not functioning.

We've fully rolled out a fix to all Slide Boxes. Boot verifications and virtualizations are fully functional again. We will likely release another fix early next week for a more permanent solution. Our apologies for any disruption this may have caused.

The hotfix has been assigned to all Slide Boxes in the field. Once upgraded, boot verifications will be run and should catch up quickly. We will continue to monitor.

We've created a hotfix release 21.2.5 and are slowly rolling it out over the course of the next hours. Once rolled out, pending boot verifications will quickly catch up. It may take a few hours for things to normalize.

Please note that we are currently experiencing a fleet-wide issue. At this time, no Slide Box can start new virtualizations, including boot verifications. Existing virtualizations (including disaster virts) are not affected.

This was caused by an automatic security upgrade to the libvirt package and our code not handling a libvirt restart correctly. These "unattended upgrades" run across our entire infrastructure to ensure all systems (including partner devices) remain fully patched and secure.

The issue is resolved by restarting the box process on the Slide Box. We are preparing a hotfix release to perform this restart fleet-wide in an efficient and controlled manner.

Please expect some friction as the day ramps up. We apologize for the disruption.