IXN POP Connectivity Outage

We are investigating a loss of resilience affecting our IXN POP.At 01:46 BST, one side of the IXN DWDM connectivity path reported both associated interfaces down. Services remain operational at this stage, but the site is operating with reduced resilience.Further updates will be provided as the investigation progresses.

The issue has been identified as a loss of both DWDM connectivity paths into the IXN POP.At 02:39 BST, the remaining DWDM path also went down, causing a loss of connectivity to the IXN POP.Initial optical diagnostics indicate receive signal loss on the affected DWDM interfaces. We are aware of unauthorised activity by a data centre engineer in our rack involving cable movement, which is considered the most likely cause.A case has been raised with DC Techs to investigate and restore the affected connectivity and outstanding resilience as soon as possible.

Connectivity to the IXN POP was restored at 02:51 BST.The confirmed full outage window was from 02:39:35 BST to 02:51:02 BST, with a total impact duration of 11 minutes and 27 seconds.Services are operational again; however, the POP remains in a reduced-resilience state while DC Techs continue work to restore the outstanding DWDM path.We are continuing to monitor the site until full resilience is restored.

We are continuing to monitor the IXN POP following the earlier restoration of connectivity.Services remain operational; however, the POP is still operating in a reduced-resilience state while we await DC Techs restoring the outstanding affected DWDM path.This has been chased with DC Techs on multiple occasions. We will also be raising this incident, along with previous issues observed at this POP, with the account manager for further review and escalation.

The outstanding IXN DWDM resilience issue appears to have been restored.At 08:04 BST, the previously affected DWDM interfaces returned to an UP state following work by data centre technicians. This indicates the downed DWDM link has been repatched/restored.IXN POP connectivity and path resilience are currently operational again. We are continuing to monitor for stability before marking this incident as resolved.