Azure Monitor - Logs Data Access Issues - Mitigated
July 06, 2023 - Started 3 months ago
- Lasted about 10 hours
Need to monitor Azure outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Azure, and never miss an outage again.
Start a Free Trial
Summary of Impact: Between at 23:15 UTC on 06 Jul 2023 and 09:00 UTC on 07 July 2023, customers using Azure Monitor may have experienced issues accessing logs, data gaps, and missed / latent alerts. Preliminary Root Cause: We determined a configuration issue triggered a bug, causing a significant unexpected increase in traffic, causing one of our backend services responsible for processing and storing logs to reach throughput limits. This caused the service's control plane to become unresponsive, leading to services becoming unable to retrieve authentication tokens. The resulting impact spread to multiple regions.Mitigation: We throttled heavy consumers and spun up additional infrastructure to process the incoming load, as well as the backlog of requests.Next Steps: We will follow up in 3 days with a preliminary Post Incident Report (PIR), which'll cover the main root cause and repair items. We'll follow that up 14 days later with a final PIR where we will share a deep dive into the incident. Stay informed about Azure service issues by creating custom service health alerts: https://aka.ms/ash-videos for video tutorials and https://aka.ms/ash-alerts for how-to documentation.