Fire incident in Singapore Availability Zone C Data Center

[Progress Update] As of 14:00 on September 16, 2024 Beijing time, the migration and restoration of the affected hardware and machineries is in progress as planned, and the remaining affected cloud products are gradually being restored. Since some of the affected hardware and machineries are located in the dangerous and blocked area of the building where access is not allowed and some hardware and machineries require to be carefully dried in order to ensure data security, hence the restoration of some long tail machines and inventories may take a longer time. For more information regarding the status of restoration, please contact us.

[Progress Update] As of 19:00 on September 14, 2024 Beijing time, some of the migrated equipment is under necessary preparation work for installation, including the drying of the equipment, wiring, powering on, verification, debugging, etc.

[Progress Update] As of 18:30 on September 13, 2024 Beijing time, the hardware equipment on the first floor is currently under safe migration. Operation and maintenance engineers have been granted access to the second floor to complete the assessment work. After the protection measures are implemented, the safe migration of the equipment on the second floor will commence.

[Progress Update] As of 17:25 on September 12, 2024 Beijing time, the operation and maintenance engineers have been granted access to the first floor area of the building. They are currently assessing conditions for safe migration and are performing emergency protection on the hardware equipment.

[Progress Update] As of 20:23 on September 11, 2024 Beijing time, the fire department is still addressing safety risks at the site, and operation and maintenance engineers are waiting for permission to enter the server building. If the on-site assessment determines that physical conditions are not suitable for restoring services on-site, the emergency team will implement the server equipment migration recovery plan.

[Progress Update] As of the early morning of September 11, 2024, most of the cloud product services affected by the network issues have been restored to normal. The remaining services in the power-interrupted server room are still waiting the physical conditions for restoration. As of 20:23 on September 11, 2024, the fire department is still addressing safety risks at the site, and operation and maintenance engineers are waiting for permission to enter the server building. If the on-site assessment determines that physical conditions are not suitable for restoring services on-site, the emergency team will implement the server equipment migration recovery plan.

[Progress Update] As of 01:46, in response to local fire department's request, due to ongoing water spraying for fire fighting, the server room has started to experience water accumulation and leaks, posing a risk of electrical short circuits. An emergency power shutdown has been implemented for one of the server building in Singapore Availability Zone C, while network services for other buildings in Availability Zone C have been gradually restored.

[Progress Update] As of 20:04, the fire alarm has not been completely eliminated. Due to fire safety control, operation and maintenance engineers are unable to enter the building in which the data centre is located, and the risk of continuously rising temperature in the data centre cannot be eliminated for the time being. Currently, some network equipment in the data centre has experienced abnormalities in high-temperature environments, affecting the network connectivity of some cloud products. If the subsequent temperature rise is not effectively controlled, the possibility of a complete network outage in Availability Zone C of Singapore will increase. If your business is deployed in Availability Zone C of Singapore, we will assist you in migrating your business as soon as possible.

[Progress Update] By 14:40, the fire situation in the affected data center has been basically brought under control, but the temperature in the facility remains high. The single availability zone (AZ) versions of some Object Storage Service (OSS), Databases, and other products will be restored after the affected physical cabinets has the conditions to restart. The remaining high availability versions of cloud products have all completed the proactive migration.

[Progress Update] As of 12:15, according to product scheduling strategy, the MaxCompute has successfully completed disaster recovery.

[Progress Update] As of 11:30, according to product scheduling strategy, the high availability versions of cloud products such as Cloud Database Redis/MongoDB/RDS MySQL, OSS, and OTS have successfully completed disaster recovery.

[Progress Update] The abnormality was caused by the explosion of lithium batteries in the Singapore data center, which led to fire and elevated temperature. Firefighters have arrived on-site to handle the situation. Most of the cloud network products and cloud security products completed the proactive switch by 10:55. Other cloud product services are still being processed. Please migrate your production workloads as soon as possible.

At 10:20 on September 10, 2024 Beijing time, Alibaba Cloud monitoring detected network access anomalies in availability zone C of the Singapore region, causing some cloud services to function abnormally. Alibaba Cloud engineers are currently handling the issue. If you have any questions, please feel free to contact us.