Outage in New Zealand Internet Exchange

[EMERGENCY OUTAGE] CHC-IX | CCE | pe1.chc1 Switch Lock Up

Resolved Major
March 26, 2023 - Started about 2 years ago
Official incident page

Need to monitor New Zealand Internet Exchange outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including New Zealand Internet Exchange, and never miss an outage again.
Start Free Trial

Outage Details

NZIX Engineers have been alerted to a management plane lock up on our switch at Christchurch Central Exchange (CCE) in CHC-IX. An emergency reboot is underway to resolve this. This bug is also impacting statistics on https://metrics.ix.nz/ for any service on this switch. Expected Downtime: ~5-10 minutes Summary of impact during this window: * Members connected to pe1.chc1 should expect their port to go offline. * ALL Members at CHC-IX should expect connectivity to rs1.chc.ix.nz to go down Due to the nature of the software fault, we cannot appropriately install BCP214 compliant filters to ports on the switch to reduce BGP disruption. This means ALL members that have routes via peers on pe1.chc1 may see a period of packet loss whilst BGP re-converges to alternate paths.
Latest Updates ( sorted recent to last )
RESOLVED about 2 years ago - at 03/26/2023 07:23PM

This incident has been resolved, newer firmware has been loaded that resolves this bug. Maintenance notices will be booked in soon to apply this code across all NZIX.

IDENTIFIED about 2 years ago - at 03/26/2023 07:12PM

NZIX Engineers have been alerted to a management plane lock up on our switch at Christchurch Central Exchange (CCE) in CHC-IX. An emergency reboot is underway to resolve this. This bug is also impacting statistics on https://metrics.ix.nz/ for any service on this switch.

Expected Downtime: ~5-10 minutes

Summary of impact during this window:
* Members connected to pe1.chc1 should expect their port to go offline.
* ALL Members at CHC-IX should expect connectivity to rs1.chc.ix.nz to go down

Due to the nature of the software fault, we cannot appropriately install BCP214 compliant filters to ports on the switch to reduce BGP disruption. This means ALL members that have routes via peers on pe1.chc1 may see a period of packet loss whilst BGP re-converges to alternate paths.

Latest New Zealand Internet Exchange outages

[OUTAGE] AKL-IX | DataVault - about 2 years ago

Be the first to know when New Zealand Internet Exchange and other third-party services go down

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3970 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook