Outage in AWS

Route 53 Resolver VPC Query Logs delayed/duplicated

Resolved Minor
September 03, 2023 - Started 9 months ago - Lasted about 2 hours

Need to monitor AWS outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including AWS, and never miss an outage again.
Start Free Trial

Outage Details

Route 53 Resolver VPC Query Logs for instances in the usw2-az1 Availability Zone in the US-WEST-2 Region from 5:30 AM PDT are not being delivered to customers. This is due to an impairment in the system that de-duplicates these logs. To mitigate this issue, we are disabling this de-duplication system, and all delayed log messages will be delivered to customers, however these will potentially include duplicate log entries. We are actively working on mitigating the impairment to the de-duplication system, and will provide an update within the next 30 minutes. This issue is only affecting query logs. All DNS queries are being answered normally, and are not impacted by this issue.
Latest Updates ( sorted recent to last )
UPDATE 9 months ago - at 09/03/2023 04:39PM

We expect all new Route 53 VPC Query Log messages to be enabled in the next 30 minutes, with potential duplicates. We have also begun processing the backlog of delayed queries, which should be fully drained within the next 4 hours.

We have not yet identified a path to mitigation of our de-duplication system, though investigations are continuing in parallel. We will provide another update within the next 30 minutes.

All DNS queries continue to be answered normally, and are not impacted by this issue.

UPDATE 9 months ago - at 09/03/2023 04:09PM

Route 53 VPC Query Log messages for new DNS queries are starting to resume delivery to customers, with potential duplicates. Once this is complete, we will begin processing the backlog of messages that were not delivered to customers, again potentially with duplicates. We have not yet identified a path to mitigation of our de-duplication system, though investigations are continuing in parallel. We will provide another update within the next 30 minutes.

All DNS queries continue to be answered normally, and are not impacted by this issue.

UPDATE 9 months ago - at 09/03/2023 03:41PM

Route 53 Resolver VPC Query Logs for instances in the usw2-az1 Availability Zone in the US-WEST-2 Region from 5:30 AM PDT are not being delivered to customers. This is due to an impairment in the system that de-duplicates these logs. To mitigate this issue, we are disabling this de-duplication system, and all delayed log messages will be delivered to customers, however these will potentially include duplicate log entries. We are actively working on mitigating the impairment to the de-duplication system, and will provide an update within the next 30 minutes.

This issue is only affecting query logs. All DNS queries are being answered normally, and are not impacted by this issue.

Easily monitor AWS and all your third-party status pages

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 3170 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook

Setup in 5 minutes or less

How much time you'll save your team, by having the outages information close to them?

14-day free trial · No credit card required · Cancel anytime