Honeycomb Outage History

Honeycomb experienced an issue where queries with certain GROUP/WHERE clauses spanning data older than 2 hours returned inconsistent results, affecting both US and EU instances for approximately 1.5-2.5 hours. No data was lost and rerunning the affected queries showed the previously missing data. The incident was resolved after 17.4 hours total duration, with the actual impact ending much earlier.

Honeycomb's activity log for the US instance experienced delays exceeding one hour due to a MySQL replica failover, affecting the logging of user activities and query runs. The service required backfilling of approximately 2.5 hours of data, with all activity log streams eventually catching up except for the query runs table which needed additional time for complete data recovery. The incident was fully resolved after monitoring confirmed all systems were operating normally.

Honeycomb experienced an issue where links in Slack triggers were not functioning properly. The problem affected users' ability to use clickable links within Slack notification triggers from the service. The issue was identified, fixed, and fully resolved within 1.6 hours.

Honeycomb's Activity Log service was interrupted for EU customers starting around 15:51 UTC, causing potential data loss during the outage period. The incident lasted 11 minutes before the service was restored to operational status. Most events were recovered once the service came back online.

Honeycomb's Activity Log experienced a 23-minute delay in processing events, preventing real-time ingestion of activity data. The service team identified and resolved the processing bottleneck, restoring normal event ingestion. No data was lost during the incident.

Honeycomb's Activity Log service in the EU region experienced delayed data ingestion starting around 19:55 UTC, affecting the timeliness of activity log events. The engineering team identified and implemented a fix within approximately 30 minutes of detection. The incident was fully resolved after 1.1 hours with all activity log data restored to live status.

Honeycomb experienced a major incident where some teams on Classic environments were unable to ingest their data through the api.honeycomb.io Event Ingest service. The engineering team identified the root cause and applied a manual fix to each affected environment while developing a systemic solution. The incident was fully resolved after 3.8 hours once the fix was uniformly applied to all impacted environments.

Honeycomb experienced a 31-hour service disruption affecting activity logs, triggers, and SLOs in the US production environment. Users encountered delays when creating or updating triggers and SLOs, along with activity log unavailability and additional SLO processing delays. The issues were progressively resolved with fixes deployed for triggers and SLOs first, followed by full recovery of all activity log data.

The Honeycomb UI experienced a major outage displaying errors to users for 13 minutes. The issue affected the main app interface at ui.honeycomb.io, preventing users from accessing the application normally. The problem was identified and resolved with a shipped fix.

Honeycomb experienced database connectivity issues that caused 0.04% of event ingest traffic to be rejected over a 20-minute period, with some users also experiencing spurious failures in the query UI. The database cluster was stabilized and the incident was resolved after 26 minutes of total impact.