Outage in Honeycomb

DNS Configuration Issue

Resolved Minor

November 30, 2023 - Started about 2 years ago - Lasted 25 minutes
Official incident page

Incident Report

In an attempt to make our DNS mechanism better and safer, we deployed a change that instead appears to have drastically dropped our ability to do DNS lookups. While we don’t have a full understanding of how that happened, we have rolled back the change and everything is back to functional. For about 12 minutes, all services were disrupted, particularly those involving queries for data. This includes triggers, and some level of our ingestion as well.

Components affected

Honeycomb api.honeycomb.io - Event Ingest Honeycomb ui.honeycomb.io - Querying Honeycomb ui.honeycomb.io - App Interface

Need to monitor Honeycomb outages?

Monitor all your external dependencies in one place
Get instant alerts when outages are detected
Be the first to know if service is down
Show real-time status on private or public status page
Keep your team informed

Start monitoring for free

Latest Updates ( sorted recent to last )

RESOLVED about 2 years ago - at 11/30/2023 08:15PM

All systems now confirmed operational

MONITORING about 2 years ago - at 11/30/2023 08:03PM

In an attempt to make our DNS mechanism better and safer, we deployed a change that instead appears to have drastically dropped our ability to do DNS lookups.
While we don’t have a full understanding of how that happened, we have rolled back the change and everything is back to functional.
For about 12 minutes, all services were disrupted, particularly those involving queries for data. This includes triggers, and some level of our ingestion as well.

Latest Honeycomb outages

Degraded querying and delayed records sent to Activity Log - 11 days ago

Querying and Ingest issues in EU - about 1 month ago

Activity log outage in production US - about 2 months ago

aws-service-degradation - 3 months ago

Delays in SLO, Service Maps processing - 3 months ago

The Status Page Aggregator with Early Outage Detection

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 5420 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook