Sydney (SYD) - Resolution Issues

While performing a configuration update of our DNS and firewall services in several locations, our Sydney location incorrectly received a configuration change that was not meant for that site, causing significant resolution issues and SERVFAIL responses. This impacted the communication between our recursive resolvers and some authoritative DNS servers. The resolution issues caused by this erroneous configuration update did not trigger an alarm in our alerting system, so this significantly delayed our awareness of the issue, as it was deployed outside of business hours in Australia, and we did not receive reports initially. We are looking into improving our monitoring and alerting to handle this specific use case. We have performed a root-cause analysis, and have updated the incorrect database values which caused Sydney to erroneously receive the aforementioned, incorrect configuration changes. We sincerely apologize for any inconvenience. Start: 12:10 UTC - August 26 End: 06:45 UTC - August 27