Outage in Honeycomb

DB migration slowing the interface down a bit

Resolved Minor
September 01, 2023 - Started over 1 year ago - Lasted about 9 hours
Official incident page

Need to monitor Honeycomb outages?
Stay on top of outages with IsDown. Monitor the official status pages of all your vendors, SaaS, and tools, including Honeycomb, and never miss an outage again.
Start Free Trial

Outage Details

A database migration has had a slightly larger impact than we expected. The interface might be slower and some triggers could take a few extra seconds to run in aggregate, but we expect everything to be fully functional and to ride it out until the end of the migration.
Latest Updates ( sorted recent to last )
RESOLVED over 1 year ago - at 09/02/2023 12:13AM

Service maps are now caught up, so we are moving this incident to resolved.

MONITORING over 1 year ago - at 09/01/2023 09:32PM

Query and application performance have returned to normal, but as a result of the incident we are behind on ingesting the events that build our service maps. We will provide a final update once service maps have caught up.

MONITORING over 1 year ago - at 09/01/2023 09:29PM

We have failed over the affected database which relieved the database load issue we encountered. Unfortunately, this caused a short period of query failures while we restarted a service that did not gracefully reconnect to the DB. Querying and performance should now be normal.

IDENTIFIED over 1 year ago - at 09/01/2023 07:47PM

We've fixed everything query side and are still seeing issues. We are currently diving deeper into our database to see what we can find out about the unexpected performance degradation. We have disabled service map updates in the meanwhile to preserve full capacity for other features.

IDENTIFIED over 1 year ago - at 09/01/2023 05:39PM

We've identified unrelated queries and specific ingest patterns that might be related to performance issues. We're still stable albeit a bit slower at times, but we're looking at whether addressing them brings more performance back.

MONITORING over 1 year ago - at 09/01/2023 04:44PM

Performance seems to be recovering and SLO/trigger performance is back within normal limits, but there is still a slightly elevated amount of query errors, including on the homepage.

MONITORING over 1 year ago - at 09/01/2023 04:17PM

The database migration has finished, but internal processes on the host have brought it to full utilization. We're investigating strategies to mitigate the effects until it catches up.

MONITORING over 1 year ago - at 09/01/2023 03:17PM

A database migration has had a slightly larger impact than we expected. The interface might be slower and some triggers could take a few extra seconds to run in aggregate, but we expect everything to be fully functional and to ride it out until the end of the migration.

All Your Service Status Pages in One Dashboard

With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.

Start free trial

No credit card required · Cancel anytime · 4000 services available

Integrations with Slack Microsoft Teams Google Chat Datadog PagerDuty Zapier Discord Webhook