Querying and Ingest issues in EU

As of 23:23UTC on December 16, the Activity Log has fully caught up and has remained up to date.

All systems are now operational.

We successfully created and migrated over to a new kafka cluster in the EU. We have resumed consuming events for our SLO and Service Maps features. No further data loss is expected for service maps. SLOs are caught up and are fully synchronized. Activity logs are still delayed and expect to be caught up in about 5 hours. We consider this incident to be mitigated, but will continue to monitor until Activity logs are caught up.

As of today, our old EU cluster is still functional, yet in a bad state. We do not consider this outage over. However, after running a full evacuation in a pre-production cluster, we are now ready to run a production version of it. This will only be in the impacted EU cluster. We are planning to do it on 6pm UTC December 16th. We expect no ingestion downtime, although we will expect delays on processing events and alerts, will suffer minor data loss in our Service Maps feature, and lose 11 days worth of Activity log events starting the night of December 5th.

We will provide updates throughout the migration.

We continue to work toward restoring the Activity Log. We’ve made meaningful progress on the underlying issues, but have had to pivot on our remediation plan due to previously unknown hard retention limits. Our engineering team is working through the next steps required to restore Activity Log processing. Next updates to come during Monday US business hours

We continue to work toward restoring the Activity Log. We’ve made meaningful progress on the underlying issues, but have had to pivot on our remediation plan due to previously unknown hard retention limits. Our engineering team is actively working through the next steps required to restore Activity Log processing.

We continue to work toward fully restoring the Activity Log. We’ve made meaningful progress on the underlying issues and continue to work through our remediation plan. Our engineering team is actively working through the next steps required to restore Activity Log processing.

We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we’ve made progress on the underlying issue and have begun the next phase of remediation. Our engineering team is pausing for the evening and will resume working through the remaining steps required to resume Activity Log processing during US business hours tomorrow. We will provide another update tomorrow morning.

We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we’ve made progress on the underlying issue and have begun the next phase of remediation. Our engineering team is actively working through the remaining steps required to resume Activity Log processing.

We continue to work toward fully restoring the Activity Log. While functionality has not yet improved, we have a clear path to recovery, and remediation work is underway. Our next steps involve coordinated infrastructure changes that will allow us to resume Activity Log processing. We’ll provide another update during US business hours Wednesday.

We are continuing to work on a fix for this issue.

We fixed the issue that was causing subsets of temporarily missing data in queries against certain partitions. We continue to work on restoring full functionality to Activity Log. We'll provide another update during US business hours Tuesday.

We continue to work on restoring full functionality to Activity Log and repair the partition causing certain customers to have subsets of unqueryable data. We will next provide an update during US business hours Tuesday.

The Activity Log is operational but is experiencing ingestion delays, which means recent activity may not be reflected immediately. We are working to restore full functionality.
Currently, a small subset of data for certain customers may not be queryable due to a partition needing repair. We are working to return the data to queryability.
After investigation of the impact starting on Friday to SLO and Trigger functionality, we're providing the update below with the details.

Overview
SLOs and Triggers were affected by Kafka and ingest outages beginning Friday, December 5 at 8:40 PM UTC. Trigger evaluations are now behaving normally except for some baseline triggers. SLOs with evaluation periods that include missing ingest or are on the affected partition will reflect that currently unqueryable data in their ongoing calculations.
SLOs
- No burn alerts were fired between December 5, 2025 at 8:40 PM UTC and December 6, 2025 at 7:15 PM UTC due to a processing delay.
- Now that ingest is restored, burn alerts should behave normally again minus a subset that are also impacted by the small amount of currently unqueryable data.
Triggers
- When ingest was down on December 6, no events were received, so all EU Trigger evaluations were impacted.
- Baseline Triggers with lookback windows that query data in the affected partition returning will also be affected by the small amount of currently unqueryable data.
- Non-baseline triggers (which can only look back 24 hours) should behave normally going forward.

We will provide an update at the end of the workday (5PM PT) or as the situation changes.

Our Kafka cluster is still operational and we are continuing to work towards restoring full reliability. We will resume work during business hours to restore full Activity Log functionality. We will continue to update as work progresses.

The Kafka cluster is still operational, but is not back to full capacity and resiliency. Work will continue tomorrow to ensure sufficient capacity, as well as during business hours to resume full Activity Log functionality. We will provide more updates tomorrow and as-needed if there are any changes to availability.

We have gotten the Kafka cluster to a stable-enough state, however it will still require remediation before the Activity Log is fully functional and it is back to a level of service sustainable until business hours. We will provide another update in 2 hours or as the situation changes.

Ingest has been fully re-enabled, and SLOs and Triggers are now up-to-date.

We have resumed ingest to api.eu1.honeycomb.io as we were able to restore partial service to the Kafka cluster.

We are continuing to investigate issues with the Kafka cluster. We will continue providing updates every 2 hours unless there are significant changes.

We are continuing to investigate issues with the Kafka cluster.

External API access is restored, event ingestion is still impacted

We have temporarily disabled event ingestion for the EU Region, in service of restoring full functionality to our EU Kafka fleet. External API access will also be disabled. Additionally, during the ongoing outage, our Service Level Objectives feature has been impacted and down since 12:30 PM Pacific time on Friday, December 5th. SLO data will not be correct until our systems catch up and we rebuild the cache

Automated systems are still working to catch up and Activity Log will remain offline until that that completes. We will post another update Saturday whether or not the Activity Log outage has been remediated.

Ingest, querying, SLOs, and Trigger alerting is back to normal. Activity Log is still impacted, we have identified the cause and are working to resolve. We are still evaluating the scope of the outage for ingest and expect to have a full answer for that posted during US business hours on Monday.
We will post at least one more update this evening.

We are continuing the investigation after business hours to stabilize ingest and determine what work will be needed to fully recover. Known impact will be updated here as the situation changes. We will post at least one more update this evening.

We have identified that 0.23% of datasets are fully affected, and a larger percentage are seeing intermittent ingestion and query failures (500s at the API level). We are also investigating a replication error in our ingestion pipeline.

A subset of customer environments may see higher than usual error rates when sending events to api.eu1.honeycomb.io, and notifications for SLOs and Triggers may be delayed for that subset. We are continuing to investigate the issue.

We are currently investigating this issue.