Outage in Honeycomb

DNS Configuration Issue

Resolved Minor

November 30, 2023 - Started over 2 years ago - Lasted 25 minutes
Official incident page

Incident Report

In an attempt to make our DNS mechanism better and safer, we deployed a change that instead appears to have drastically dropped our ability to do DNS lookups. While we don’t have a full understanding of how that happened, we have rolled back the change and everything is back to functional. For about 12 minutes, all services were disrupted, particularly those involving queries for data. This includes triggers, and some level of our ingestion as well.

Components affected

Honeycomb api.honeycomb.io - Event Ingest Honeycomb ui.honeycomb.io - Querying Honeycomb ui.honeycomb.io - App Interface

Trusted by 1000+ teams

Need to monitor Honeycomb outages?

Stop finding out about outages from your users. Monitor 6,020+ cloud services and get alerted the moment something breaks.

Ready to take control?

Start Free Trial

No credit card required

Set up in 2 minutes

14-day free trial

Latest Updates ( sorted recent to last )

RESOLVED over 2 years ago - at 11/30/2023 08:15PM

All systems now confirmed operational

MONITORING over 2 years ago - at 11/30/2023 08:03PM

In an attempt to make our DNS mechanism better and safer, we deployed a change that instead appears to have drastically dropped our ability to do DNS lookups.
While we don’t have a full understanding of how that happened, we have rolled back the change and everything is back to functional.
For about 12 minutes, all services were disrupted, particularly those involving queries for data. This includes triggers, and some level of our ingestion as well.

Latest Honeycomb outages

Ingest issues for some customers - about 4 hours ago

Activity log delay in production US - 1 day ago

Honeycomb UI displaying an error - 23 days ago

Degraded Ingest due to database connectivity issues - about 1 month ago

Query Errors - about 2 months ago

The Status Page Aggregator with Early Outage Detection

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 6020 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook