Honeycomb experienced a major incident where some teams on Classic environments were unable to ingest their data through the api.honeycomb.io Event Ingest service. The engineering team identified the root cause and applied a manual fix to each affected environment while developing a systemic solution. The incident was fully resolved after 3.8 hours once the fix was uniformly applied to all impacted environments.
Trusted by 1,000+ teams
Stop finding out about outages from your users. Monitor 6,320+ cloud services and get alerted the second something breaks.
We can confirm that the fix was applied uniformly to all impacted and potentially impacted environments; no teams should see ingest errors related to this issue anymore.
We are continuing to work on the systemic solution and monitor. We'll leave the incident open until we're able to confirm the issue won't recur.
The manual fix has been applied to all environments.
We've applied the manual fix to nearly all affected datasets (less than 1% remaining). The systemic solution is still in progress.
We've identified a manual fix and are deploying that by hand to each affected environment, while also investigating a systemic solution.
We have further narrowed down that only teams on Classic environments are currently seeing issues. We are figuring out mitigation mechanisms at the moment.
We have confirmed reports of some teams getting issues ingesting their data. We have identified a probable source behind this behavior and are currently trying to correct it.
With IsDown, you can monitor all your critical services' official status pages from one centralized dashboard and receive instant alerts the moment an outage is detected. Say goodbye to constantly checking multiple sites for updates and stay ahead of outages with IsDown.
Start free trialNo credit card required · Cancel anytime · 6320 services available
Integrations with