Outage in AWS

Route 53 Resolver VPC Query Logs delayed/duplicated

Resolved Minor
September 03, 2023 - Started about 2 years ago - Lasted about 2 hours

Incident Report

Route 53 Resolver VPC Query Logs for instances in the usw2-az1 Availability Zone in the US-WEST-2 Region from 5:30 AM PDT are not being delivered to customers. This is due to an impairment in the system that de-duplicates these logs. To mitigate this issue, we are disabling this de-duplication system, and all delayed log messages will be delivered to customers, however these will potentially include duplicate log entries. We are actively working on mitigating the impairment to the de-duplication system, and will provide an update within the next 30 minutes. This issue is only affecting query logs. All DNS queries are being answered normally, and are not impacted by this issue.

Need to monitor AWS outages?

One place to monitor all your cloud vendors. Get instant alerts when an outage is detected.

Try IsDown risk-free 14-day free trial · No credit card required
Latest Updates ( sorted recent to last )
UPDATE about 2 years ago - at 09/03/2023 04:39PM

We expect all new Route 53 VPC Query Log messages to be enabled in the next 30 minutes, with potential duplicates. We have also begun processing the backlog of delayed queries, which should be fully drained within the next 4 hours.

We have not yet identified a path to mitigation of our de-duplication system, though investigations are continuing in parallel. We will provide another update within the next 30 minutes.

All DNS queries continue to be answered normally, and are not impacted by this issue.

UPDATE about 2 years ago - at 09/03/2023 04:09PM

Route 53 VPC Query Log messages for new DNS queries are starting to resume delivery to customers, with potential duplicates. Once this is complete, we will begin processing the backlog of messages that were not delivered to customers, again potentially with duplicates. We have not yet identified a path to mitigation of our de-duplication system, though investigations are continuing in parallel. We will provide another update within the next 30 minutes.

All DNS queries continue to be answered normally, and are not impacted by this issue.

UPDATE about 2 years ago - at 09/03/2023 03:41PM

Route 53 Resolver VPC Query Logs for instances in the usw2-az1 Availability Zone in the US-WEST-2 Region from 5:30 AM PDT are not being delivered to customers. This is due to an impairment in the system that de-duplicates these logs. To mitigate this issue, we are disabling this de-duplication system, and all delayed log messages will be delivered to customers, however these will potentially include duplicate log entries. We are actively working on mitigating the impairment to the de-duplication system, and will provide an update within the next 30 minutes.

This issue is only affecting query logs. All DNS queries are being answered normally, and are not impacted by this issue.

The Status Page Aggregator Built for IT Teams

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4522 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook