[MS/DNS] Investigating issue

CyberFOX DNS Incident Update

Yesterday at 10:00 AM EST, we identified a routing configuration issue following a recent upstream network optimization. This issue caused traffic to not be evenly balanced across cluster instances as intended. As a result, several clusters responsible for DNS filtering became overutilized, reaching 90%+ capacity, which led to slower DNS resolution for some users and intermittent resolution failures for others.

Our team immediately began mitigation efforts by manually adding additional resources to the affected clusters. While this rebalance was in progress, we temporarily shifted a portion of traffic from US East to US West, which helped stabilize performance while the corrected configuration was being deployed.

10:30 AM EST: Traffic levels returned to normal in our monitoring, and we began gradually shifting traffic back to US East.
12:10 PM EST: All traffic and DNS resolution performance were fully restored and operating normally.

Preventative Measures

To help prevent similar issues in the future, we have implemented additional monitoring and alerting on our load balancers and cluster instances to detect imbalance and elevated utilization earlier.

We appreciate your patience and understanding while we worked through this issue. As always, thank you for trusting CyberFOX DNS.

We’re currently investigating reports of elevated latency affecting one of our regions. The CyberFOX DNS team is actively working to identify the cause and restore normal performance as quickly as possible. We’ll share updates as more information becomes available. Thanks for your patience!