Ip Transit and Internet degraded performance in NSW

Impact Summary
At approximately 09:49 Monday 7th December 2020 AEDST, for up to 120 minutes, FireNet observed disruptions to a number of transit and peering connections within our New South Wales edge devices causing an internet outage for a majority of services.

Technical Brief
FireNet engineers received alerts from internal systems for our edge devices notifying high cpu and memory usage. Upon investigation we noticed all BGP sessions in a hung state where they held current routes but were not updating. The devices were accepting commands but not applying. We were able to re-route traffic to other nodes and have now identified the issue that affected the nodes.

Moving Forward
FireNet engineers have sent through logs of the affected devices to the vendor to identify the root cause. Once the logs have been analysed, we will work with the vendor to test and deploy changes to these nodes.
FireNet will notify of any maintenance in due course.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We have identified that a number of IP transit and Internet services in NSW are experiencing degraded performance.

Our carrier is currently investigating the issue.

Further updates to follow.


Please contact the FireNet NOC should you have any questions, comments, or concerns.
Email: support@firenet.com.au
Phone one of the following numbers:
Australia: 1300 636 636
International customers please call +61 2 9397 7050