Network incident after routine reboots

The incident in the MCO1 region has been fully resolved. We've taken corrective measures to prevent a recurrence. All systems are now fully operational and stable. We appreciate your patience and understanding during this incident.

We have resolved the connectivity issue in the MCO1 region. The root cause was identified as a discrepancy in the nftables rules definitions between different nodes. We have corrected the rules on the affected node and have taken measures to prevent further similar issues. The region is now fully operational. We continue to monitor the situation and we are applying the necessary fixes to other nodes as a preventative measure. We appreciate your understanding during this incident.

We have identified an issue with one of our nodes in the MCO1 region during a routine reboot operation. The node failed to recover, causing some connectivity issues within the region. The error was due to an installed version of iptables on the affected node, which interfered with our system's internal mechanisms. After performing some fixes, connectivity is now being restored. We are monitoring the situation closely to ensure full recovery.

Please note that event ingestion and processing has not been impacted by this incident, only frontend and API access.

Thank you for your patience.