ryzen03 outage

The system is now fully operational again. Preliminary indications suggest that the issue was likely caused by either a faulty cable or transceiver. We are continuing to run additional diagnostics to confirm the root cause and ensure long-term stability.

We have received feedback from firstcolo confirming that the issue could not be resolved by replacing the network components (NIC, transceiver, and fiber). The connectivity problem persists, and the node remains isolated from the network.

firstcolo’s engineering team has now escalated the case and will proceed with a full on-site diagnostic workflow to further investigate the root cause at the hardware and infrastructure level. This includes in-depth testing of the server’s network subsystem, switch-side configuration, and any potential backplane.

We will provide another update as soon as we receive additional information from the data center.

As of 12:00 CET today, the compute node ryzen03, part of our new AMD Ryzen 9 9950X product lineup, unexpectedly powered down. After performing detailed debugging and telemetry analysis, we have determined that the system is currently not receiving any network traffic whatsoever—neither from internal infrastructure nor from external upstream sources.

The installed 10G NIC is detected normally by the OS and reports no apparent faults, but the node remains completely unreachable due to the absence of active network ingress and egress. To mitigate this as rapidly as possible, we have escalated the issue to our data center operations team. An on-site technician has been dispatched, and as a precautionary measure we have requested a full component swap, including:

Transceiver
Fiber patch cable
10G network interface card

This is to ensure the node can be restored to service as quickly and reliably as possible. We are currently awaiting updates from the technician.

We sincerely apologize for the service interruptions affecting ryzen01 and ryzen03. All other host systems in the AMD Ryzen 9 9950X product series are operating nominally, and we remain fully prepared to provision additional compute capacity as needed.

Further updates will follow as soon as new information becomes available.